Facial image recognition using pseudo-images

ABSTRACT

This disclosure relates to the use of “pseudo-images” to perform image recognition, e.g., to perform facial image recognition. In an embodiment, the pseudo-image is obtained by starting with a real world image and, after optional preprocessing, subjecting the image to a non-linear transformation that converts the image into a pseudo-image. While real world objects (or, more generally, real world patterns) may be perceivable in the starting image, they cannot be perceived in the pseudo-image. Image recognition takes place by comparing the pseudo-image with a library of known pseudo-images, i.e., image recognition takes place in pseudo-image space without a return to real world space. In this way, robust image recognition is achieved even for imperfect real world images, such as, real world images that have been degraded by noise, poor illumination, uneven lighting, and/or occlusion, e.g., the presence of glasses, scarves, or the like in the case of facial images.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of co-pending U.S.application Ser. No. 16/453,545 filed Jun. 26, 2019, the contents ofwhich in its entirety is hereby incorporated by reference. Thisapplication also claims the benefit under 35 USC § 119(e) of U.S.Provisional Application No. 62/693,136 filed on Jul. 2, 2018, thecontents of which in its entirety is hereby incorporated by reference.

GOVERNMENT FUNDING

This invention was made with government support under grant number R01DC014701 awarded by the National Institutes of Health. The government hascertain rights in the invention.

FIELD

This disclosure relates to computer systems and computer-implementedmethods for performing image recognition. In a particularly importantembodiment, the systems and methods are used to identify a human subjectwithin an image through facial recognition. More generally, thedisclosure provides computer-implemented methods and computer systemsfor processing images composed of pixels or, more generally, imagescomposed of components, to find objects, patterns, or features embeddedin the images that can be used for classification, identification, orother purposes.

BACKGROUND

Facial recognition by computer devices has broad applications, not onlyin historically important areas such as national security and thecriminal justice system, but also in recent years in commercial settingsand social media interactions. Consequently, facial recognition has beenand continues to be the subject of intense study with various scientificjournals being dedicated to the problem (e.g., the International Journalof Computer Vision and the IEEE publications entitled IEEE Transactionson Image Processing and IEEE Transactions on Pattern Analysis andMachine Intelligence) and with international conferences being regularlyconvened to report on progress being made (e.g., the annual conferencesof the International Association for Pattern Recognition and the IEEEComputer Society Conferences on Computer Vision and Pattern Recognitionand the IEEE International Conference on Automatic Face and GestureRecognition).

Success in facial recognition has been achieved in laboratoryenvironments. However, it remains a challenge in practical applications,where images are often collected from imperfect sources such assurveillance cameras, the internet, or mobile devices. Facial imagesobtained under such real world conditions are often degraded by noise,poor illumination, uneven lighting, and/or occlusion, making itdifficult to recognize the person or persons whose faces appear in theimage. Variation in facial expression, pose, and camera angles oftenpresent additional difficulties. A central challenge of facialrecognition is thus to achieve robust and invariant recognition of thesame face under varying conditions and with degraded images. Such imageswill be referred to herein as “imperfect images.”

Computerized facial recognition is usually performed using systems thatrepresent faces in some form, following which a matching process isconducted. Some of the systems that achieve successful recognitioninvolve extracting features from images using either learning-basedapproaches or manually-curated features. Representations of the imagesare then subject to matching that often involves statistical approachesto classify and/or identify the faces. While manually-curated featurescan provide robustness, it is prohibitive to enumerate all possiblefeatures. Deep learning approaches can allow a system to perform featureextraction with supervised or unsupervised algorithms. However, theygenerally require a large training set that covers numerous possibleconditions. If there are conditions not included in the training set,the system likely will fail to perform in practice.

SUMMARY AND GENERAL DESCRIPTION

The present disclosure is directed to the above problems in facial imagerecognition. More generally, the disclosure is directed to providingrobust methods for processing images composed of components (e.g.,pixels) to find objects, patterns, or features embedded therein.

In accordance with a first aspect, the disclosure provides a method ofperforming computer-implemented image recognition comprising:

(a) providing a first-image having M components;

(b) providing a predetermined transformation matrix, wherein:

-   -   (i) the predetermined transformation matrix is an M×K matrix in        which the K columns constitute a set of K dictionary elements,        and    -   (ii) the predetermined transformation matrix is constructed by a        method comprising performing a sparse, non-negative        factorization of an M×N matrix in which the N columns constitute        a set of N training images, each training image having M        components; and

(c) constructing a pseudo-image for the first-image using thepredetermined transformation matrix to perform a sparse, non-negativetransformation of the first-image, said pseudo-image for the first-imageconsisting of K element weights, each element weight being for one ofthe K dictionary elements;

wherein the M×N matrix has a rank R and K satisfies one or both of thefollowing relationships:

(i) K is greater than or equal to M; and

(ii) K is greater than or equal to R.

As is known, the row rank of a matrix is the number of rows of thematrix that are linearly independent and the column rank is the numberof columns that are linearly independent. The row rank always equals thecolumn rank and thus the number of linearly independent rows or columnsconstitutes a matrix's “rank.” A matrix is a “full rank matrix” if itsrank equals the largest possible rank for a matrix of the samedimensions, i.e., if the rank of the matrix is the lesser of the numberof rows and columns.

The M×N matrix will often be a full rank matrix and it will alsonormally have N>M. Under these conditions, R equals M so that the secondrelationship becomes the same as the first relationship, i.e., K≥M.Thus, in many cases, the above relationships are equivalent to thepredetermined transformation matrix being a square matrix (K=M) or, moretypically, a rectangular matrix with more columns than rows (K>M.

In certain embodiments of the first aspect of the disclosure, the methodfurther comprises the following steps (d) and (e):

(d) providing a set of S pseudo-images of known images, each of said Spseudo-images consisting of K element weights, each element weight beingfor one of the K dictionary elements; and

(e) comparing the pseudo-image of step (c) with the set of Spseudo-images of step (d) to, for example, determine a likelihood thatthe first-image is one or more of:

-   -   (i) in one or more classes or categories of the known images,    -   (ii) a particular known image,    -   (iii) not in one or more classes or categories of the known        images, and    -   (iv) not a known image.

In other embodiments of the first aspect of the disclosure, the methodcomprises the further step of reporting the results of the comparison ofstep (e) or otherwise using those results, e.g., using the results ofthe comparison to control the operation of a robot in a machine visionapplication of the disclosed method.

In accordance with a second aspect, the disclosure provides a method ofperforming computer-implemented image recognition comprising:

(a) providing a first-image having M components;

(b) providing a predetermined transformation matrix, wherein:

-   -   (i) the predetermined transformation matrix is an M×K matrix in        which the K columns constitute a set of K dictionary elements,        and    -   (ii) the predetermined transformation matrix is constructed by a        method comprising performing a sparse, non-negative        factorization of an M×N matrix in which the N columns constitute        a set of N training images, each training image having M        components;

(c) constructing a pseudo-image for the first-image using thepredetermined transformation matrix to perform a sparse, non-negativetransformation of the first-image, said pseudo-image for the first-imageconsisting of K element weights, each element weight being for one ofthe K dictionary elements;

(d) providing a set of S pseudo-images of known images, each of said Spseudo-images consisting of K element weights, each element weight beingfor one of the K dictionary elements; and

(e) comparing the pseudo-image of step (c) with the set of Spseudo-images of step (d) to, for example, determine a likelihood thatthe first-image is one or more of:

-   -   (i) in one or more classes or categories of the known images,    -   (ii) a particular known image,    -   (iii) not in one or more classes or categories of the known        images, and    -   (iv) not a known image.

In certain embodiments of the second aspect of the disclosure, themethod comprises the further step of reporting the results of thecomparison of step (e) or otherwise using those results, e.g., using theresults of the comparison to control the operation of a robot in amachine vision application of the disclosed method.

In accordance with a third aspect, the disclosure provides a method ofperforming computer-implemented image recognition comprising:

(a) providing a first-image having M components each of which has onlyone of two possible values;

(b) providing a predetermined transformation matrix, wherein:

-   -   (i) the predetermined transformation matrix is an M×K matrix in        which the K columns constitute a set of K dictionary elements,        and    -   (ii) the predetermined transformation matrix is constructed by a        method comprising performing a sparse, non-negative        factorization of an M×N matrix in which the N columns constitute        a set of N training images, each training image having M        components each of which has only one of said two possible        values; and

(c) constructing a pseudo-image for the first-image using thepredetermined transformation matrix to perform a sparse, non-negativetransformation of the first-image, said pseudo-image for the first-imageconsisting of K element weights, each element weight being for one ofthe K dictionary elements.

In certain embodiments of the third aspect of the disclosure, the methodfurther comprises the following steps (d) and (e):

(d) providing a set of S pseudo-images of known images whose componentshave only one of said two possible values, each of said S pseudo-imagesconsisting of K element weights, each element weight being for one ofthe K dictionary elements; and

(e) comparing the pseudo-image of step (c) with the set of Spseudo-images of step (d) to, for example, determine a likelihood thatthe first-image is one or more of:

-   -   (i) in one or more classes or categories of the known images,    -   (ii) a particular known image,    -   (iii) not in one or more classes or categories of the known        images, and    -   (iv) not a known image.

In other embodiments of the third aspect of the disclosure, the methodcomprises the further step of reporting the results of the comparison ofstep (e) or otherwise using those results, e.g., using the results ofthe comparison to control the operation of a robot in a machine visionapplication of the disclosed method.

With regard to step (a) of the first, second, and third aspects of thedisclosure, the first-image can be an image obtained from, for example,a digital imaging device, e.g., a standalone digital camera or a digitalcamera embodied in another device, e.g., a cell phone. Such images areexamples of “original-images” as defined below. In certain embodiments,the first-image can be an original-image that has been subjected to oneor more levels of preprocessing. To facilitate the presentation, thefirst-image of step (a) is referred to below as a“first-image-of-interest” or as a “step(a)-first-image” in order todistinguish the first-image of step (a) from other first-images that areused in other portions of the overall process, e.g., first-images thatare used as training images and first-images that are used in thepreparation of pseudo-image libraries.

With regard to step (b) of the first, second, and third aspects of thedisclosure, the predetermined transformation matrix used in this stepcan be thought of as the “engine” of the disclosed image recognitionprocess. In an embodiment, the sparse, non-negative factorization usedin obtaining the predetermined transformation matrix employs at leastone Frobenius norm. (As used herein, a Frobenius norm of a matrix is thesquare root of the sum of the squares of the components of the matrix.)Importantly, the matrix of training images used in obtaining thepredetermined transformation matrix (referred to herein as the “trainingset of images” or simply the “training set”) need not include thefirst-image-of-interest and typically will not include it. That is, thedisclosed method is able to perform image recognition on images thatwere not part of the method's training set. This is an importantadvantage of the process because, among other things, it allows imagerecognition to be performed on imperfect images that were not part ofthe training set, including images that suffer from, for example, one ormore of noise, corruption, or occlusion. In an embodiment, onceconstructed, the predetermined transformation matrix is stored in anon-transitory, computer-readable medium for later use.

To facilitate the presentation, the pseudo-image of step (c) is referredto below as a “pseudo-image-of-interest” or as a “step(c)-pseudo-image”in order to distinguish the pseudo-image of step (c) from otherpseudo-images that are used in other portions of the overall process,e.g., pseudo-images that are generated during the production of thepredetermined transformation matrix and pseudo-images that are used inthe preparation of pseudo-image libraries. In an embodiment, onceconstructed, the step(c)-pseudo-image is stored in a non-transitory,computer-readable medium for later analysis and/or use. In anembodiment, the sparse, non-negative transformation used in constructingpseudo-images (other than the pseudo-images generated during theproduction of the predetermined transformation matrix) employs at leastone Lz norm. (As used herein, an Lz norm of a vector is the square rootof the sum of the squares of the components of the vector.)

With regard to optional steps (d) and (e) of the first and third aspectsof the disclosure and required steps (d) and (e) of the second aspect ofthe disclosure, in an embodiment, the set of pseudo-images of knownimages used in these steps (the “library of pseudo-images” or simply the“library”) is obtained using the same predetermined transformationmatrix as used in step (c). Importantly, in step (e), the comparison isbetween pseudo-images, not between first-images. It is this comparisonof pseudo-images as opposed to a comparison of first-images that is akey element in providing the disclosed process with its improvedrobustness compared to prior techniques for performing imagerecognition.

Upon completion of the comparison of step (e), the results of thecomparison can be reported directly to the user or stored for subsequentuse, reporting, or analysis. When the comparison of step (e) is used forclassification, the reporting may be as simple as identifying a singlecategory and/or a single class for the first-image-of-interest (and thusthe original-image; see below). For example, in the case of facialrecognition, the classification can be as basic as categorizing thefirst-image-of-interest as being a male face or a female face. Thereporting will typically be more detailed, e.g., it will typicallyprovide information regarding multiple categories and/or multipleclasses of interest. Optionally, the reporting can include an indicationof the confidence level of the classification for one, more than one, orall of the categories or classes for which a comparison was performed.

When the comparison of step (e) is used for identification, thereporting may be as simple as notifying the user that a “match” has beenfound. Typically, the notification will be accompanied by at least thename of the known image. Usually, in addition to the name, the reportingwill include other relevant data regarding the known image, as well as acopy of the known image. Optionally, the reporting can include anindication of the confidence level of the identification, e.g., thereporting can include a similarity or comparison score. In anembodiment, the indication of confidence can include copies of one ormore known images with lower confidence levels (lower likelihoods ofcorresponding to the first-image-of-interest) than the known image withthe highest confidence level.

As noted above, an important feature of the image recognition methoddisclosed herein is that once in pseudo-image space, the method remainsin pseudo-image space and does not return to first-image space to, forexample, perform the comparison of step (e). In this way, the robustnessof the method, e.g., its ability to handle imperfect images, issignificantly improved. Robustness has also been found to depend on thevalue of K, with larger values of K leading to more robustness but atthe expense of longer computation times and/or larger storagerequirements.

In particular, as will be discussed in more detail below, it has beenfound that robustness increases with the ratios of K to M and R. (Asnoted above, in many cases, R will be equal to M.) Quantitatively, incertain embodiments, one or both of the K/M and K/R ratios are greaterthan or equal to 1.0, or greater than or equal to 2.0, or greater thanor equal to 3.0, or greater than or equal to 4.0, or greater than orequal to 5.0.

As noted above, the M×N matrix will often be a full rank matrix and itwill also normally have N>M. Under these conditions, R equals M so thatthe K≥R criterion for robust image recognition becomes K≥M, i.e., for afull rank M×N matrix with N>M, the dimension of the pseudo-image in Kspace needs to be greater than or equal to the dimension of thefirst-image in M space. Under these circumstances, the transformationfrom a first-image to a pseudo-image using the predeterminedtransformation matrix can be thought of as an “expansion” or“decompression” of the first-image from M components to K components.That is, the predetermined transformation matrix takes an image with agiven number of components (M components) and transforms it (expands itor decompresses it) into an image with more components (K elementweights). A priori one would not think that this would be helpful inperforming image recognition. In accordance with the present disclosure,just the opposite has been found—the transformation is extremelyeffective in performing image recognition and provided the expansion issufficiently large, results in high levels of robustness in performingimage recognition on imperfect images.

On its face, the M→K decompression strategy of the image recognitiontechniques disclosed herein goes against the conventional wisdom thatimage processing should achieve data compression, not expansion.However, because of the sparseness condition applied during thegeneration of pseudo-images, the decompression in most cases does notmean that more storage is needed for the pseudo-image than for thefirst-image since relatively few of the K element weights making up thepseudo-image will have values that need to be stored to represent thepseudo-image. That is, many and, in most cases, most of the elementweights will be zero or essentially zero and thus all that needs to bestored are the values and locations in the pseudo-image of the elementweights that are not zero or not essentially zero. Accordingly, in mostcases, the image recognition techniques disclosed herein simultaneouslyachieve both effective image recognition and reduced storagerequirements. (Although it will typically not affect storagerequirements, it can be noted that because the transformation from thefirst-image to the pseudo-image is a non-negative transformation, all ofthe element weights that are stored are positive numbers.)Quantitatively, designating the number of element weights that need tobe stored as K′, the ratio of K′ to M will generally be less than 1.0,or less than or equal to 0.75, or less than or equal to 0.50, or lessthan or equal to 0.25, or less than or equal to 0.10, or less than orequal to 0.05, or equal to 1/M.

In the case of grayscale first-images, having K≥M and/or K≥R has beenfound to be a base requirement for robust image recognition. Forfirst-images where the components of the image can only have one of twovalues, e.g., on or off, as opposed to many values, e.g., 256 values asin an 8-bit grayscale, it has been found that acceptable levels ofrobustness can be achieved through the use of pseudo-images incomparison step (e) even if neither of the K≥M and K≥R relationships aresatisfied. Example 10 and, in particular, FIGS. 33 and 34, illustratethis difference between 2-value images (binary images) compared tograyscale images. Specifically, in this example, acceptably robust imagerecognition of imperfect images of symbols (specifically, letters andcharacters) was achieved both when K was greater than M(FIG. 33) andwhen it was less than M(FIG. 34). As illustrated in this example, havingK greater than M makes the process substantially more robust, but therobustness is sufficient for practical applications when K is less thanM and the first-image is a binary image. The same results are found whenK is compared with R.

In accordance with a fourth aspect of the present disclosure, the valueof K and/or the value of its ratio to one or both of M and R is varieduntil a suitable level of robustness is identified for the particularimage recognition problem being addressed. Examples 1-8 below illustratethis aspect of the disclosure where a K/M ratio of 0.8 was foundsufficient for performing facial recognition on faces that had not beensubjected to facial modifications but insufficient for faces that hadbeen subjected to modification. A K/M ratio of 2.4, on the other hand,succeeded in providing correct identifications both for unmodified andmodified faces, and a K/M ratio of 4.0 was even better.

In accordance with a fifth aspect, the disclosure providesnon-transitory, computer-readable media and computer systems forperforming the image recognition methods disclosed herein. Thenon-transitory, computer-readable media, which can be sold and/ordistributed as articles of commerce, can contain computer instructions(computer code) capable of being executed on a computer system toperform part or all of the disclosed image recognition techniques.

In accordance with a sixth aspect, the disclosure provides one or moredatasets of pseudo-images for use in steps (d) and (e) as pseudo-imagelibraries. The dataset or sets can be contained in non-transitory,computer-readable media that are sold and/or distributed as articles ofcommerce. Likewise, one or more predetermined transformation matricesfor use in step (b) and/or one or more training sets for obtainingpredetermined transformation matrices can be contained innon-transitory, computer-readable media that are sold and/or distributedas articles of commerce. The distribution can, for example, be over theinternet which, among other things, can facilitate updating ofpseudo-image libraries to, for example, add new pseudo-images or removepseudo-images no longer relevant to the image recognition beingperformed. The non-transitory, computer-readable media can be in the“cloud” or at a user's location.

Additional aspects of the present disclosure are set forth below underthe heading “Features of the Disclosure.”

A preferred application of the image recognition techniques disclosedherein is facial recognition. Other applications include objectrecognition and symbol recognition (machine reading). More generally,the disclosed image recognition techniques can be used in all forms ofmachine vision. Non-limiting examples of the variety of images that canbe analyzed using the technology disclosed herein, as well asnon-limiting examples of applications for the technology, are discussedbelow under the heading “Industrial Applicability.”

Additional properties and advantages of the technology disclosed hereinare set forth in the detailed description that follows, and in part willbe readily apparent to those skilled in the art from that description orrecognized by practicing the technology as described herein. Theaccompanying drawings are included to provide a further understanding ofthe technology, and are incorporated in and constitute a part of thisspecification. It is to be understood that the various aspects of thetechnology disclosed in this specification and in the drawings can beused individually and in any and all combinations. It is also to beunderstood that the general description set forth above and the detaileddescription which follows are merely exemplary of the invention and areintended to provide an overview or framework for understanding thenature and character of the invention as defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an original-image of a person-of-interest.

FIG. 2A shows the original-image-of-interest of FIG. 1 after a firstround of first-level preprocessing, specifically, cropping.

FIG. 2B shows the original-image-of-interest after a second round offirst-level preprocessing, specifically, down-sampling. The image ofthis figure was used as a first-image-of-interest; it contained 625pixels (M=625) in a 25×25 array.

FIG. 3 shows five first-images of a training set of 2,000 first-images(N=2,000). The first-images were obtained using the same first-levelpreprocessing as that used in obtaining the first-image-of-interest ofFIG. 2B.

FIG. 4 shows a portion of the dictionary elements of a predeterminedtransformation matrix obtained using the full training set from whichthe portion of FIG. 3 was taken. In this instance, each pseudo-image has1,500 element weights (K=1,500) for 1,500 dictionary elements, oneweight for each dictionary element. Thirty-six of the 1,500 dictionaryelements are shown. Each dictionary element has 625 components shown asa 25×25 two-dimensional array in this figure.

FIG. 5 shows a portion of a library (S=2,000) of pseudo-images of thetraining set. Five pseudo-images corresponding to each of thefirst-images shown in FIG. 3 are displayed.

FIG. 6 shows the pseudo-image-of-interest for thefirst-image-of-interest of FIG. 2B. It also shows the dictionaryelements for the pseudo-image and highlights the dictionary elementswith the largest element weights. The pseudo-image-of-interest wasobtained using the predetermined transformation matrix some of whosedictionary elements are shown in FIG. 4. All 1500 dictionary elementsand their element weights are displayed.

FIG. 7 shows significant element weights (coefficient values) for thepseudo-image-of-interest of FIG. 6. The coefficient value of eachelement is indicated by the height of the line corresponding to theelement.

FIG. 8 shows the similarity scores of the pseudo-image-of-interest ofFIG. 6 with the pseudo-image library obtained from the 2,000first-images of the training set. Similarity was measured by cosinedistances. The figure is an example of the reporting of theidentification of a first-image-of-interest and thus anoriginal-image-of-interest by a comparison of a pseudo-image-of-interestwith a pseudo-image library. The first-image of the face within thetraining set that has the highest similarity score is displayed. It isidentical to the first-image-of-interest.

FIG. 9 shows a portion of the dictionary elements of a predeterminedtransformation matrix obtained using the full training set from whichthe portion of FIG. 3 was taken. In this instance, each pseudo-image has500 element weights (K=500) for 500 dictionary elements. Thirty-six ofthe 500 dictionary elements are shown.

FIG. 10 shows a portion of a library of pseudo-images of the trainingset when transformed into pseudo-images of 500-dimension. Fivepseudo-images corresponding to each of the first-images shown in FIG. 3are displayed.

FIG. 11 shows the pseudo-image-of-interest for thefirst-image-of-interest of FIG. 2B for K equal to 500. Thepseudo-image-of-interest was obtained using the predeterminedtransformation matrix some of whose dictionary elements are shown inFIG. 9. Each element weight (each component) of the pseudo-image is fora dictionary element of the predetermined transformation matrix, i.e.,each element weight of the pseudo-image is for a column of thepredetermined transformation matrix. The dictionary elements with thetop (largest) twelve element weights are highlighted in the bottom panelof this figure. The grayscale of each element weight in the pseudo-imageindicates the magnitude of the element weight. All 500 element weightsand dictionary elements are displayed.

FIG. 12 shows the twelve significant dictionary elements of thepseudo-image of FIG. 11. The element weight (coefficient value) of eachof the significant dictionary elements is indicated by the height of theline corresponding to the element. Larger pictures of the twelvedictionary elements are displayed in the bottom panel.

FIG. 13 shows the reporting of the identification of thefirst-image-of-interest and thus the original-image by a comparison ofthe pseudo-image-of-interest of FIG. 11 with the full pseudo-imagelibrary from which the portion of FIG. 10 was taken. The first-image ofa face within the pseudo-image library that has the highest similarityvalue is displayed.

FIG. 14 shows the identification of a face in a pseudo-image library butwith a pair of glasses that changes and somewhat obscures the originalface. The face with glasses was not in the pseudo-image library, but theprocess correctly identified it as the exact person without the glasses.K equaled 1,500 for this figure.

FIG. 15 shows the identification of a face in a pseudo-image library butwith a pair of sunglasses that completely obscures the eyes of thesubject. The face with sunglasses was not in the pseudo-image library,but the process correctly identified it as the exact person without thesunglasses. K equaled 1,500 for this figure.

FIG. 16 shows the identification of faces with expressions differentfrom the ones in the pseudo-image library. The top panel shows a personwhose smiling face was in the training set. Her non-smiling face (the“new” face) was properly identified even though it was not in thetraining set and thus not in the pseudo-image library prepared from thetraining set. The bottom panel shows a person whose non-smiling face wasnot in the training set and thus not in the pseudo-image libraryprepared from the training set. Her smiling face was correctlyidentified. K equaled 1,500 for this figure.

FIG. 17 illustrates the correct identification of a face with differentfacial modifications. The face with sunglasses (top left), the facedisguised with a mustache (top right), and the face with both sunglassesand a mustache (bottom left), each generated a pseudo-image that whencompared with the pseudo-image library which contained the unadulteratedface gave the highest similarity score for the unadulterated face. Kequaled 1,500 for this figure.

FIG. 18 illustrates the identification of a female face when differentparts of the face were occluded. The faces with high similarity scoresincluded the original face to be identified. Although not wishing to bebound by any particular theory of operation, it is believed that femalefaces can be more difficult to identify than male faces and consequentlythe face of the pseudo-image library with the highest similarity scorewas not the correct face in two of the four cases. K equaled 1,500 forthis figure. As shown in FIG. 27, when K was increased to 2,500, thecorrect face was identified in all four cases.

FIG. 19 illustrates the identification of a male face when differentparts of the face were occluded. The face with the highest similarityscore was the original face to be identified in all cases. K equaled1,500 for this figure.

FIG. 20 shows the identification of a face in a pseudo-image library butwith a pair of glasses added to the face as in FIG. 14. K equaled 500for this figure, instead of 1,500 as in FIG. 14. In contrast to FIG. 14,a wrong face was identified.

FIG. 21 shows the identification of a face in a pseudo-image library butwith a pair of sunglasses obscuring the eyes of the woman's face as inFIG. 15. K equaled 500 for this figure, instead of 1,500 as in FIG. 15.In contrast to FIG. 15, a wrong face was identified.

FIG. 22 shows the identification of two faces in a pseudo-image librarybut with different facial expressions as in FIG. 16. K equaled 500 forthis figure, instead of 1,500 as in FIG. 16. In contrast to FIG. 16,wrong faces were identified.

FIG. 23 shows the identification of a face in a pseudo-image library butwith added facial accessories, including a pair of sunglasses, amustache, or both, as in FIG. 17. K equaled 500 for this figure, insteadof 1,500 as in FIG. 17. In contrast to FIG. 17, wrong faces wereidentified.

FIG. 24 shows the identification of a female face in a pseudo-imagelibrary but with different parts of her face obscured as in FIG. 18. Kequaled 500 for this figure, instead of 1,500 as in FIG. 18. In contrastto FIG. 18, wrong faces were identified in all cases.

FIG. 25 shows the identification of a male face in a pseudo-imagelibrary but with different parts of his face obscured as in FIG. 19. Kequaled 500 for this figure, instead of 1,500 as in FIG. 19. In contrastto FIG. 19, wrong faces were identified.

FIG. 26 shows the identification of a face in a pseudo-image library butwith added facial accessories, including a pair of sunglasses, amustache, or both, as in FIG. 17 and FIG. 23. K equaled 2,500 for thisfigure and the correct face was identified with a substantially highersimilarity score in all cases compared to other faces in the library.

FIG. 27 shows the identification of a female face in a pseudo-imagelibrary but with different parts of her face obscured as in FIGS. 18 and24. K equaled 2,500 for this figure and the correct face was identifiedwith a substantially higher similarity score in all cases compared toother faces in the library.

FIG. 28 shows a set of faces that were not in the training set used toproduce the pseudo-image library of Example 1. The faces were from theYale face image database. Faces for 15 different individuals with 11different facial expressions and lighting conditions for each face areshown in this figure.

FIG. 29 illustrates the identification of the faces shown in FIG. 28using the predetermined transformation matrix of Example 1. The graphshows the pairwise similarity scores between pairs of pseudo-images forthe faces from the Yale database of FIG. 28. Dark grayscales indicatehigh similarity scores. Faces belong to the same person are grouped andindicated by the number along the axes. The graph shows that despitedifferent facial expressions and lighting conditions, faces belonging tothe same person are highly similar in their pseudo-images, but not thosebelonging to different persons.

FIG. 30 illustrates the pairwise similarity scores between thepseudo-images for faces from the Yale database of FIG. 28 andpseudo-images for the faces of the training set of Example 1. Incontrast to the similarity of the pseudo-images of the Yale facesbelonging to the same person, the Yale faces exhibit little similarityto those in the training set.

FIG. 31 shows 1,000 symbols (letters and characters) that were used as atraining set for image recognition in accordance with the techniquesdisclosed herein. Each symbol was digitized as a 16×16 first-image.

FIG. 32 illustrates the generation of pseudo-images for the symbols ofthe training set of FIG. 31 and the similarity scores of thepseudo-images for the capital letter “H” and a Chinese “bing” characterwith those pseudo-images used as a pseudo-image library. K equaled 800for this figure.

FIG. 33 illustrates the identification of the capital letter “H” and aChinese “bing” character when the letter and the character werecorrupted by missing pixels (shown next to the original symbol). Kequaled 800 for this figure. The similarity scores between thepseudo-images for the corrupted letter/character and each of thepseudo-images in the pseudo-image library are shown. The correctletter/character was identified.

FIG. 34 illustrates the identification of the capital letter “H” and aChinese “bing” character when the letter and the character werecorrupted by missing pixels (shown next to the original symbol). Thecorruption was the same as in FIG. 33. K equaled 100 for this figure.The correct symbols were identified but many pseudo-images now have highsimilarity score values.

FIG. 35 shows the cosine error rates (1−cosine distances) using K equalto 800 (top panel) vs. K equal to 200 (bottom panel) for images composedof varying numbers of pixels randomly selected from an original image.The drop in error rate with increasing numbers of pixels is faster forthe larger K value.

FIG. 36 shows the first-images (N=1,000) used in comparing the de novoand sequential approaches for creating an augmented predeterminedtransformation matrix. Each of these first-images is a 16×16 array ofbinary pixels (M=256). The first images of this figure are the same asthose of FIG. 31, but with different formatting.

FIG. 37 shows the dictionary elements (K=1,000) produced using the denovo approach for creating an augmented predetermined transformationmatrix. Each of the dictionary elements is a 16×16 array of binarypixels (M=256).

FIG. 38 shows the dictionary elements (K=1,000) produced using thesequential approach for creating an augmented predeterminedtransformation matrix. Each of the dictionary elements is a 16×16 arrayof binary pixels (M=256).

FIG. 39 shows cosine similarity between the dictionary elements of FIG.37 produced by de novo learning and the dictionary elements of FIG. 38produced by sequential learning.

FIG. 40 shows pairwise correlations between the components of thedictionary elements of FIG. 38 produced by sequential learning.

FIG. 41 shows pairwise correlations between components of pseudo-imagesfor the first-images of FIG. 36 where the pseudo-images were produced bysequential learning.

FIG. 42 is a flowchart illustrating an exemplary embodiment of thepresent disclosure which produces a predetermined transformation matrixand a set of pseudo-images for a training set.

FIG. 43 is a flowchart illustrating an exemplary embodiment of thepresent disclosure which produces a pseudo-image-of-interest from afirst-image-of-interest and compares the pseudo-image-of-interest with apseudo-image library.

FIG. 44 is a flowchart illustrating an exemplary embodiment of thepresent disclosure which produces a pseudo-image from a known image andincorporates the pseudo-image in one or more pseudo-image libraries.

FIG. 45 is a flowchart illustrating an exemplary embodiment of thepresent disclosure for determining whether a predeterminedtransformation matrix is in need of augmentation.

FIG. 46 is a flowchart illustrating an exemplary embodiment of thepresent disclosure for augmenting a predetermined transformation matrixby the sequential approach. The updating of Φ_(i) and A_(i) can beperformed using, for example, Algorithm 3 below.

FIG. 47 is a functional block diagram illustrating components of arepresentative example of a computer system for use in practicingembodiments of the methods of the disclosure.

TERMINOLOGY AND NOTATION

The following terms and their associated singulars and plurals and thefollowing notation are used in the specification and in the claims.

(A) Original-Image and First-Level, Second-Level, and OtherPreprocessing

An “original-image” is an ordered sequence of components (e.g., pixels),each component having a value and a location within the sequence.Typically, a human will be able to perceive one or more objects from thereal world in an original-image (e.g., a human face in the case offacial recognition), but not necessarily in all cases, e.g., anoriginal-image can be a computer-generated image such as a UPC or matrixbarcode. An original-image can be generated by a digital imaging device,e.g., a standalone digital camera or a digital camera embodied inanother device, e.g., a cell phone. An original-image can also be ananalog image that has been digitized.

In the case of original-images that are in color, the image can bebroken down into composite colors, e.g., the red, green, and bluesubcomponents (e.g., subpixels) of an image produced by a digitalcamera, and each of the composite colors processed as a separateoriginal-image. Alternatively, the composite colors can be concatenatedinto a single original-image. For example, in the case of a 25×25 RGBcolor image, the concatenated original image would have 1,875components, rather than 625 components if the image were not in color.

If desired, an original-image can be subjected to “first-levelpreprocessing” prior to being transformed into a pseudo-image. Unlikesecond-level preprocessing discussed below, first-level preprocessingmaintains the relative relationships between the components of an imageboth in terms of the relative values and the relative locations of thecomponents in the sequence of components. Examples of first-levelpreprocessing include, but are not limited to, reduction in the numberof components (down-sampling), cropping of portions of the image, affinetransformations of the image, such as, rotating, translating,stretching, and/or contracting all or part of the image, normalization,and the like.

As with an original-image, a first-level preprocessed image is anordered sequence of components, each component having a value and alocation within the sequence. In the case of original-images in whichhumans can perceive one or more objects from the real world, one or moreobjects from the real world will normally still be perceivable by humansafter first-level preprocessing unless all such objects are removed by,for example, cropping. The number of components in a first-levelpreprocessed image can be the same as or less than the number ofcomponents in the image from which it is derived.

For some applications, second-level preprocessing can be performed on anoriginal-image or a first-level preprocessed image (referred to as the“starting-image” for the second-level preprocessing). In an embodiment,second-level preprocessing can comprise performing a Fourier transformof the starting-image and using the resulting Fourier coefficients asthe components of the first-image. Along these same lines, wavelettransforms can be used and the resulting wavelet coefficients used asthe components of the first-image. These types of pre-processing canproduce a “reduced-image” that is an ordered sequence of components,each component having a value and a location within the sequence, wherethe number of components in the reduced-image is less than the number ofcomponents in the starting-image. A reduction in the number ofcomponents reduces storage requirements and depending on the number ofcalculations needed to generate the reduced-image, can reduce overallprocessing times.

Like an original-image or a first-level preprocessed image, asecond-level preprocessed image is an ordered sequence of components,each component having a value and a location within the sequence.However, unlike first-level preprocessing, second-level preprocessingdoes change the relationship between the components. The change in therelationship between components can be in terms of relative values,relative locations of the components in the sequence of components, orboth relative values and relative locations in the sequence ofcomponents. Consequently, in the case of images that contain objectsfrom the real world, humans may find it more difficult or, in somecases, impossible to perceive those objects in images that haveundergone second-level preprocessing.

Although first-level and second-level preprocessing have been definedseparately to facilitate the presentation, in practice, the two types ofpreprocessing can be integrated with one another as a singlepreprocessing step in the overall image recognition process. Also, eachof first-level and second-level preprocessing steps can be performedmultiple times in any order. In the case of images that contain objectsfrom the real world, when first-level preprocessing is performed on animage that has undergone second-level preprocessing, humans may find itmore difficult or impossible to perceive the real world objects in theresulting second-level-followed-by-first-level preprocessed image.

In addition, preprocessing that is not specifically characterizable asfirst-level preprocessing or second-level preprocessing can be used. Forexample, low-pass, high-pass, or band-pass spatial filtering can changethe relative values between components. Such filtering can produceimages in which humans may find it easier, rather than harder, toperceive objects from the real world because, for example,high-frequency noise has been removed in the case of low-pass filteringor edges of objects have been enhanced in the case of high-passfiltering. Preprocessing thus includes, but is not limited to,first-level and second-level preprocessing.

(B) First-Image

A “first-image” is an original-image or an original-image that has beensubjected to preprocessing. Thus, a first-image is an ordered sequenceof components (e.g., pixels), each component having a value and alocation within the sequence. As discussed above, for a typicaloriginal-image, a human will be able to perceive one or more objectsfrom the real world (e.g., a human face) in the original-image. For afirst-image that is the original-image or the original-image that hasundergone only first-level preprocessing, this will continue to be thecase. For other types of preprocessing, humans may find it difficult orimpossible to perceive objects from the real world in a first-image.

In the discussion below, the first-image is treated as a vector, with anindividual first-image being represented by x_(n) and a set offirst-images arranged as the columns of a matrix being represented by X.The number of components in a first-image (the “dimension” of thefirst-image) is represented by the letter M and the number of columns(number of first-images) in the matrix X is represented by the letter N.Thus, in the above notation, n can have a value between 1 and N, eachx_(n) has M components, and X is an M×N matrix.

Although in the discussion below, first-images and other orderedsequences of components are treated as vectors (one-dimensional arrays),in a computer, ordered sequences of components can be stored and/orprocessed as higher dimensional arrays, e.g., two or three dimensionalarrays, if desired.

(C) Predetermined Transformation Matrix and Dictionary Elements

A “predetermined transformation matrix” is a matrix having M rows and Kcolumns. Each of the K columns is referred to as a “dictionary element”or simply an “element.” Each dictionary element is an ordered sequenceof M components, each component having a value and a location within thesequence. Thus, if desired, each dictionary element can be displayed asan image (see, for example, FIG. 4). In the discussion below, apredetermined transformation matrix is represented by the matrix Φ.

As described above and discussed in more detail below, a predeterminedtransformation matrix is obtained by a method comprising performing asparse, non-negative factorization of a matrix of training images. Asdiscussed below, sparseness both with regard to the production of thepredetermined transformation matrix and the production of pseudo-imagesfrom first-images can be measured using an L₁ norm, an L₀ norm, or anL_(p) norm where p is greater than zero and less than one, the L₁ normbeing preferred in most cases. Other measures of sparseness can also beused (see below). The matrix of training images is a matrix of knownimages, specifically, a matrix of known first-images. Those knownfirst-images may have been subjected to preprocessing. In such a case,in an embodiment, the first-image-of-interest (step(a)-first-image) canbe subjected to the same preprocessing as the first-images making up thetraining set. In other embodiments, the preprocessing used to producethe first-image-of-interest may be different from the preprocessing usedto produce the first-images of the training set (or thefirst-image-of-interest may be an original-image with no preprocessing)provided that, notwithstanding the different preprocessing, thepreprocessed first-image-of-interest and the preprocessed first-imagesmaking up the training set have the same number of components (i.e., thesame M value).

The factorization produces the predetermined transformation matrix andpseudo-images for the training images. Both the predeterminedtransformation matrix and the pseudo-images are constrained to benon-negative. The pseudo-images are also constrained to be sparse. Thepredetermined transformation matrix, however, is not constrained to besparse. Thus, in the phrase “a sparse, non-negative factorization,”non-negativity applies to both the predetermined transformation matrixand the pseudo-images that are produced by the factorization, whilesparseness applies only to the pseudo-images. The process of producingthe predetermined transformation matrix is non-linear, as opposed tolinear, as a result of, among other things, the non-negativityconstraints that can be considered analogous to rectification which isan inherently non-linear process. Likewise, the production ofpseudo-images from first-images using the predetermined transformationmatrix is also a non-linear process.

Using the matrix notation adopted above for first-images, the matrix oftraining images can be represented by a matrix X. In this notation, apredetermined transformation matrix Φ is a matrix having M rows and Kcolumns obtained by performing a sparse, non-negative factorization ofan M row by N column matrix X. In an embodiment, the factorizationemploys a minimization of at least one Frobenius norm. In the discussionbelow, an individual pseudo-image produced using the predeterminedtransformation matrix is represented by a, while a set of pseudo-imagesarranged as columns in a matrix is represented by A. Using thisnotation, the relationship of the matrix X of training images used inproducing the predetermined transformation matrix Φ and the set ofpseudo-images A for the training images produced during the process ofdetermining Φ can be written:

X=ΦA,

where A has K rows and N columns. Thus, the predetermined transformationmatrix can be thought of as performing a transformation from a basishaving M dimensions (the number of rows of the X matrix) to a basishaving K dimensions (the number of rows of the A matrix).

Although not wishing to be bound by any particular theory of operation,it is believed that the combination of the non-negativity constraints onΦ and A in combination with the sparseness constraint on A force theinclusion in Φ of key features of the images of the training set, e.g.,eye shape in the case of facial images, thus making Φ robust in terms ofimage recognition. The use of Frobenius norms can also contribute to therobustness of the technique. Because such norms are squared norms, theycan be considered as related to “energy,” which, in turn, can beconsidered as related to “information content.” The key features are thefeatures with the most information content and by using Frobenius norms,Φ can be forced to capture those features. The non-negativity constraintthen prevents dilution of the key features with non-key features, andthe sparseness constraint binds key features together in the dictionaryelements. The result is that the dictionary elements can have specificreceptor fields, i.e., they can capture particular shapes andcombinations of shapes in the images of the training set, and by havinga sufficient number of dictionary elements, robust image recognition canbe achieved.

Suitable values for M(the number of components in the first-images), N(the number of first-images in the training set), and K (the number ofelement weights for dictionary elements in the pseudo-images) to achieverobustness can be readily determined by skilled workers based on thepresent disclosure. In general terms, the number (N) of first-images inthe training set scales with the number (M) of components in thefirst-images, i.e., one usually needs a larger training set as thenumber of components (e.g., number of pixels) in the first-imagesincreases. Also in general terms, Nis approximately equal to K andnormally is greater than K.

As discussed above, K itself depends on M and/or R with, in certainembodiments, K satisfying one or both of the relationships K/M≥1.0 andK/R≥1.0, where, as above, R is the rank of the X matrix. Moreparticularly, as also discussed above, in certain embodiments, one orboth of the K/M and K/R ratios are greater than or equal to 2.0, orgreater than or equal to 3.0, or greater than or equal to 4.0, orgreater than or equal to 5.0. These ratios are particularly important inthe case of grayscale images and can be relaxed in the case of binaryimages if desired. Further, in general, the value of K scales with N,i.e., as N increases, it is generally desirable for K to increase.However, increasing K also increases the demands on computationalresources including computational (CPU) times and/or memoryrequirements. Also, the effectiveness of increases in K can diminishwhen K becomes too large. Thus, as will be evident to a skilled person,trade-offs in terms of accuracy and computational costs are made whenchoosing values of K, M, and N for any particular application of theimage recognition techniques disclosed herein.

As discussed above and more fully below, K is the dimension of apseudo-image obtained from an M-dimensional first-image using thepredetermined transformation matrix and thus to achieve robustness, inan embodiment, the dimension of the pseudo-image will be greater thanthe dimension of the first-image when the M×N matrix is full rank andN>M, except in the case of binary images where the dimension of thepseudo-image can be smaller than the dimension of the first-image ifdesired. The M×N matrix will usually not be far from full rank, i.e., Rwill not be much smaller than M, and thus satisfying K≥M will not besignificantly more stringent than satisfying K≥R. While thetransformation from M-space to K-space will not formally be an expansionin the number of dimensions used in capturing the information content ofthe first-image when K is greater than or equal to R but less than M,the transformation will not be a significant compression. The same willbe true in the case of binary images where the transformation maysomewhat reduce the number of dimensions but usually not to a greatextent. As noted above, the M→K decompression strategy of the imagerecognition techniques disclosed herein goes against the conventionalwisdom that image processing is based on data compression, notexpansion. Likewise, merely a small compression goes against theconventional wisdom.

(D) Pseudo-Image and Element Weights

A “pseudo-image” is an ordered sequence of K components, each componenthaving a value and a location within the sequence. Thus, if desired, apseudo-image can be displayed as an image (see, for example, thegrayscale pseudo-images of FIGS. 5 and 6). In practice, i.e., forpurposes of computer coding, pseudo-images can be treated as vectors.

As discussed above, a pseudo-image is obtained from a first-image by anon-linear transformation of the first-image, specifically, a sparse,non-negative transformation of the first-image performed using thepredetermined transformation matrix discussed above. Because thetransformation applies a non-negativity constraint, the value of each ofthe K components of a pseudo-image is zero or a positive number.

Each of the K components functions as an “element weight” for one of theK dictionary elements (K columns) of the predetermined transformationmatrix used in constructing the pseudo-image from the first-image.Because the transformation from the first-image to the pseudo-imageapplies a sparseness constraint on the pseudo-image, normally there areonly a few components (element weights) with larger values and manycomponents (element weights) with smaller or zero values (see, forexample, FIGS. 6 and 7). In practice, only the components with largervalues need be stored and used for the comparison of apseudo-image-of-interest with pseudo-images of known images.Accordingly, the term “pseudo-image” includes the case where thepseudo-image as stored and/or used comprises all K components and thecase where the pseudo-image comprises less than all K components, thesmaller components not being explicitly included in the pseudo-image asstored and/or used.

When displayed as a two-dimensional array, humans do not perceive apseudo-image as showing objects from the real world (see, for example,the grayscale pseudo-images of FIGS. 5 and 6). This is so even in caseswhere humans could perceive objects from the real world in thefirst-image prior to the transformation of the first-image into thepseudo-image, i.e., even if the first-image was an original-image or anoriginal-image that had undergone only first-level preprocessing (see,for example, the grayscale first-images of FIGS. 5 and 6). This is animportant distinction of the present technique in that theclassification and/or identification of images is performed using imagesthat do not contain humanly-perceivable objects. Among other things,this avoids the storage of humanly-perceivable images of specificindividuals and thus avoids the privacy issues associated with suchstorage.

(E) Pseudo-Image Library

A “pseudo-image library” is a set of pseudo-images used forclassification and/or identification of a pseudo-image-of-interest (astep(c)-pseudo-image) obtained from a first-image-of-interest (astep(a)-first-image).

(F) Classification and Identification

“Classification” is associating an image with a set having at least twomembers with one or more common characteristics, e.g., a class orcategory, while “identification” is associating an image with a sethaving one member, e.g., an individual.

DETAILED DESCRIPTION AND PREFERRED EMBODIMENTS

FIGS. 1-8 illustrate an embodiment of the image recognition techniquesof the present disclosure. In particular, they illustrate an embodimentin which the techniques are applied to the problem of facialrecognition. The specific procedures employed in producing the figuresare discussed below and in Example 1.

The figures are introduced at this point in the description to providecontext for the general discussion that follows. It is to be understoodthat the use of facial recognition as a vehicle for explaining thetechnology should not be considered limiting and that the pseudo-imagetechniques disclosed herein are equally applicable to other imagerecognition problems such as those involving object recognition, symbolrecognition, and the like (see the Industrial Applicability sectionbelow for other non-limiting applications of the technology disclosedherein).

The aspects of the disclosed image recognition technique illustrated inFIGS. 1-8 are as follows:

-   -   (1) FIG. 1 shows an original-image of a person of interest.    -   (2) FIGS. 2A and 2B show the original-image of FIG. 1 after two        rounds of first-level preprocessing, specifically, FIG. 2A shows        the original image after cropping and FIG. 2B shows it after        down-sampling. The image of FIG. 2B is the        first-image-of-interest (step(a)-first-image) which is        subsequently transformed into a pseudo-image, specifically, the        step(c)-pseudo-image.    -   (3) FIG. 3 shows a portion of a training set of first-images        obtained using the same two rounds of first-level preprocessing        as that used in obtaining the first-image-of-interest of FIG.        2B.    -   (4) FIG. 4 shows a portion of the dictionary elements of a        predetermined transformation matrix obtained using the full        training set from which the portion of FIG. 3 was taken.    -   (5) FIG. 5 shows a portion of a library of pseudo-images.    -   (6) FIG. 6 shows the pseudo-image-of-interest for the        first-image-of-interest of FIG. 2B. The pseudo-image-of-interest        was obtained using the predetermined transformation matrix some        of whose dictionary elements are shown in FIG. 4.    -   (7) FIG. 7 shows the dictionary elements with the largest        element weights (coefficient values) of the        pseudo-image-of-interest of FIG. 6.    -   (8) FIG. 8 shows the reporting of the identification of the        first-image-of-interest and thus the original-image by a        comparison of the pseudo-image-of-interest of FIG. 6 with the        full pseudo-image library from which the portion of FIG. 5 was        taken.

As discussed above, the present disclosure employs pseudo-images toperform image recognition. The process begins with an original-image(see, for example, FIG. 1) that is typically subjected to at least somepreprocessing (e.g., cropping and down-sampling) to produce afirst-image-of-interest (see, for example, FIG. 2B). Alternatively, theoriginal-image can be used directly as the first-image-of-interestwithout preprocessing. As discussed above in the Terminology andNotation section, the first-image-of-interest is convenientlyrepresented as a vector x.

The first-image-of-interest is transformed into apseudo-image-of-interest (see, for example, FIGS. 6 and 7) using apredetermined transformation matrix composed of dictionary elements(see, for example, FIG. 4) obtained using a training set of first-images(see, for example, FIG. 3). As discussed above in the Terminology andNotation section, the pseudo-image-of-interest is convenientlyrepresented by a vector a, the predetermined transformation matrix by amatrix Φ, the training set of first-images, i.e., the collection of theset of x vectors for the training images, by a matrix X, and thepseudo-images for the training set by a matrix A.

Classification and/or identification of the first-image-of-interest andthus the original-image is then performed by comparing thepseudo-image-of-interest with a library of pseudo-images (see, forexample, FIG. 5). The results of the comparison can be reported to auser by, for example, displaying a known image that corresponds to theentry in the library for which a match was found or, in the case ofclassification, an identifier for a class or category. The results ofthe comparison can include an indication of the likelihood that theclassification and/or identification is accurate, e.g., an indication ofthe likelihood that the first-image-of-interest corresponds to a class,a category, or an individual. The indication can be a similarity orcomparison score (see, for example, FIG. 8).

The predetermined transformation matrix is obtained by a methodcomprising performing a sparse, non-negative factorization on a matrixof vectorized first-images (the training set). The size of the trainingset will depend on the classification and/or identification to beperformed. For example, in the case of facial recognition, if theclassification and/or identification is to be performed on a limited setof individuals, e.g., individuals who are to be permitted access to aparticular facility, then a relatively small training set may besufficient provided enough individuals are included in the set so thatthe predetermined transformation matrix is able to classify and/oridentify the limited set of individuals and distinguish them fromindividuals not in the limited set. At the other extreme, classificationand/or identification of individuals in the general population will, ingeneral, require a large training set so that enough features areembedded in the predetermined transformation matrix to perform theclassification and/or identification. A suitable size for the trainingset can be readily found for any particular application of the imagerecognition techniques disclosed herein by routine experimentation basedon the present disclosure.

In certain embodiments, image recognition on imperfect images isperformed with limited and, in some cases, no imperfect images in thetraining set. Specifically, it has been found that image recognition onimperfect images and, in particular, facial recognition on imperfectfacial images can be performed without the need to purposely includelarge numbers of imperfect images in the training set. Imperfect imagescan be included in the training set if desired and, in some cases, alimited number of imperfect images in the training set may be useful.For example, inclusion in the training set of imperfect images where theimperfection is, for example, pose and/or facial expression can make thepredetermined transformation matrix more robust in terms ofclassification and/or identification in some cases.

Unlike prior image recognition techniques and, in particular, priorfacial recognition techniques, large numbers of imperfect images are nota requirement for successful image recognition. The lack of such arequirement permits the use of smaller training sets than wouldotherwise be needed. Such smaller training sets, in turn, improve thespeed and/or storage requirements of the process used to generate thepredetermined transformation matrix. These higher speed and/or smallerstorage considerations also apply to pseudo-image libraries whether thelibrary is based on the training set, the training set plus additionalpseudo-images, or a set of pseudo-images that excludes the training set(see below). Likewise, the higher speed and/or smaller storageconsiderations apply to comparisons of a pseudo-image-of-interest withone or more pseudo-image libraries.

As discussed above, in some embodiments, the first-image-of-interest isan original-image that has been subjected to preprocessing, where thepreprocessing can be first-level preprocessing, second-levelpreprocessing, or a combination of first-level and second-levelpreprocessing. When preprocessing is to be used, the same preprocessingis preferably performed on the training set prior to its use inproducing the predetermined transformation matrix. While suchpreprocessing commonality is preferred because it can provide improvedimage recognition, it is not a requirement for successful imagerecognition. This lack of a requirement of common preprocessing betweenthe first-images of the training set and the first-image-of-interest canbe beneficial in many situations. Specifically, it provides flexibilityto the overall process by permitting a given training set to be usedwith first-images-of-interest that have been subjected to varyingdegrees of preprocessing.

The pseudo-image-of-interest for the first-image-of-interest is obtainedusing the predetermined transformation matrix to perform a sparse,non-negative transformation of the first-image-of-interest. Thefactorization that generates the predetermined transformation matrixalso generates pseudo-images for the members of the training set. Incertain embodiments, classification and/or identification of theoriginal-image can be performed by comparing thepseudo-image-of-interest with the pseudo-images for the members of thetraining set. In such a case, the pseudo-images for the members of thetraining set functions as a pseudo-image library for performingclassification and/or identification.

The pseudo-image-of-interest can also be used for purposes other thanclassification and/or identification. For example, if thefirst-image-of-interest is for a person, object, symbol, or the likewhose category, class, or identity is known but who is not already partof a pseudo-image library, then the pseudo-image-of-interest can be usedto augment one or more libraries, i.e., the pseudo-image-of-interest canbe added to one or more libraries. The thus expanded library orlibraries can then be used in the future for classification and/oridentification of first-images-of-interest and thus original-images. Inthis way, pseudo-image libraries can become more valuable over time forthe classification and/or identification of images.

In some embodiments, multiple pseudo-image libraries can be used in theclassification and/or identification process, including libraries ofdifferent sizes. The classification and/or identification process cancompare the pseudo-image-of-interest with all of the libraries or with asubset of libraries. For example, the comparison process can proceedthrough the libraries in a selected order, e.g., from the smallestlibrary to the largest library, until a match having a sufficient levelof confidence is found whereupon the comparison process can be ended.

In some embodiments, the pseudo-image library need not includepseudo-images for the members of the training set, i.e., the library canexclude some or all of the members of the training set. For example, inconnection with facial recognition, this can be the case where thetraining set provides sufficient variability in facial features so as toproduce a predetermined transformation matrix capable of extractingfacial features from a variety of individuals irrespective of whetherthose individuals are in the training set.

In such embodiments, whether for facial recognition or other types ofimage recognition, the training set can be thought of as seeding thepredetermined transformation matrix with the ability to producepseudo-images-of-interest containing sufficient information to performclassification and/or identification. Once the predeterminedtransformation matrix is well-seeded, the training set can be viewed ashaving served its purpose and thus no longer being needed for thecomparison step. As a specific example in the field of criminal law, atraining set could be composed of individuals who do not have criminalrecords and the pseudo-image library could include only individuals withcriminal records.

The foregoing are just a few non-limiting examples of the wide varietyof pseudo-image libraries that can be used in the practice of the imagerecognition techniques disclosed herein. In general terms, thepseudo-image library or libraries used in the comparison step will beadjusted to meet the needs of particular image recognition situations.Adjustments of the library or libraries can also take place over time asneeds change. Thus, pseudo-images can be added or subtracted, andlibraries can be combined with one another or subdivided into partsbased on initial or subsequent needs. Among the parameters that can beconsidered in selecting a library or set of libraries for any particularapplication are accuracy (level of confidence) of image recognitionachieved with the library or libraries, comprehensiveness of the libraryor libraries, process speed, and memory requirements. As is typical,trade-offs will often be needed between these competing considerations.

Comparison of a pseudo-image-of-interest with one or more pseudo-imagelibraries can be performed in a variety of ways. For example, Euclideandistances can be calculated between a pseudo-image-of-interest and thepseudo-images of a library, with smaller distances being indicative ofcorrespondence between the pseudo-image-of-interest and particularpseudo-images of the library. As another example, cosine similarityvalues (scores) can be calculated, i.e., cos(θ) values can becalculated, where θ is the angle between the pseudo-image-of-interestand a particular pseudo-image of the library, both treated as vectors.When the pseudo-image-of-interest is aligned or nearly aligned with aparticular pseudo-image of the library, θ equals zero or nearly zero, sothat the cosine similarity value is 1.0 or close to 1.0, thus indicatinga correspondence between the pseudo-image-of-interest and the particularpseudo-image of the library.

Whatever measure or measures are used, because the pseudo-images of thelibrary are for known images, the result of the comparison can, forexample, be used to determine whether the first-image corresponding tothe pseudo-image-of-interest is one or more of:

-   -   (i) in one or more classes or categories of the known images,    -   (ii) a particular known image,    -   (iii) not in one or more classes or categories of the known        images, and    -   (iv) not a known image.

The results of the comparison can be employed in various ways. One basicuse is to provide a user with a visual, oral, or other type ofnotification that a “match” has or has not been found. The notificationwill typically be accompanied by a report which may be as simple as thename of the known image or may include other data including anindication of the level of confidence of the identification. The reportcan be in visual, oral, or other form. In the case of machine vision,the result of the comparison may be a set of instructions for executionby, for example, a robot, e.g., instructions to interact with theidentified object in a particular way. Other ways in which the result ofthe comparison can be used will be evident to persons skilled in the artfrom the present disclosure.

Various algorithms can be used to obtain a predetermined transformationmatrix Φ by factorization of a matrix X of training images. Similarly,various algorithms can be used to transform a first-image-of-interest xinto a first-pseudo-image-of-interest a. The following are non-limitingexamples of suitable algorithms that can be used.

Algorithms for Generating a Predetermined Transformation Matrix Using aTraining Image Set

A key to the robustness in pattern recognition of the disclosed methodis the predetermined transformation matrix. As discussed above, thepredetermined transformation matrix is obtained from a set offirst-images that are used as a training set. The process of generatingthe predetermined transformation matrix comprises the factorization of amatrix containing the training set of first-images into two separatematrices. The factorization of a matrix into two separate matrices is anapproach generally characterized as blind source separation (BSS), whichhistorically was developed to identify or approximate independentsources of signals. General discussions of BSS can be found in Comon andJutten 2010 and Yu, Hu et al. 2014. The methods disclosed herein employlinear algebra, including operations on matrices and (column) vectorsand solutions to systems of linear equations, general discussions ofwhich can be found in Gill, Murray et al. 1991 and Strang 2006. Themethods also employ optimization techniques, general discussions ofwhich can be found in the literature (Gill, Murray et al. 1991, Dantzigand Thapa 1997, Chen, Donoho et al. 2001, Boyd and Vandenberghe 2004,Candes and Tao 2005, Donoho 2006, Comon and Jutten 2010, Donoho, Tsaiget al. 2012, Yu, Hu et al. 2014).

In general terms, the generation of the predetermined transformationmatrix can be achieved through a two-step process. First, each image inthe training set that is not already digitized is digitized and asneeded preprocessed into a first-image of dimension m₁×m₂, e.g.,25×25=625. As discussed above, the preprocessing is preferably the sameas will be performed on original-images which are to undergo imagerecognition. The first-image is further represented as an M-dimensionalvector (M=m₁·m₂) so that each first-image can form a column vector ofthe training set matrix X. For a training set consisting of Nfirst-images (e.g., N faces), the training set matrix X is, therefore,an M×N dimension matrix.

In the second step, the matrix X is factorized into two matrices A andΦ. Here, Φ is the predetermined transformation matrix. The dimension ofΦ is M×K. A is a K×N matrix, which represents the N first-images in Kdimensions. Each column of A is the transformation of the correspondingfirst-image of the training set into its pseudo-image, the dimension ofthe pseudo-image being K.

Using the training set of first-images, the predetermined transformationmatrix is generated in such a way that the pseudo-images correspondingto the first-images of the training set are sparse. Note that thepseudo-image is not unique but depends on the images making up thetraining set, as well as the initialization step of A and Φ, as inAlgorithm 1 below. However, once Φ has been determined, the pseudo-imagegenerated using Φ (e.g., the pseudo-image generated using Algorithm 2below) is only dependent on Φ and is independent of the initializationused in the pseudo-image generation process, e.g., as discussed below inconnection with Algorithm 2, the initialization used in the pseudo-imagegeneration process can be, for example, random.

In the method disclosed herein, two restrictions are imposed on the BSSproblem. First, all elements in Φ and A are required to be non-negative.Second, A is required to be sparse. These constraints are important inobtaining the predetermined transformation matrix that is used togenerate pseudo-images. In particular, these constraints are importantfor robust image recognition. The sparsity constraint results ingrouping of distinguishing morphological features of first-images intodictionary elements, such that first-images with different morphologiesdo not share the same significant coefficients. The non-negativityconstraint enforces the grouping together of features that occurtogether in the training set. This is achieved by preventing the use ofnegative coefficients which can cause features to be subtracted out ofcomplex feature combinations. The non-negativity constraint thusprevents the dictionary elements from becoming overly complicated, i.e.,it prevents complex feature combinations that do not co-occur in theactual images from remaining in the analysis as could occur if negativecoefficients were permitted. A consequence of these two constraints isthat features that are likely to occur together in first-images getextracted into a few dictionary elements, which bind the co-occurringand therefore defining feature combinations in the first-images, intodistinct dictionary elements. This arrangement thus maximizes thedistinction between dictionary elements for these features anddictionary elements for other features that likewise tend to occurtogether in first-images.

In other words, these two constraints are highly effective inclassifying (aggregating) source features based on statistical relationsbetween them. For example, a particular contour of a nose can end upbeing in one dictionary element with certain cheek features, while theshape of an ear plus eyebrow features are in another, based on the facesused in the training set. If, for example, all of the faces in thetraining set happened to have substantially identical say, ears, thenthe sparsity constraint would tend to drive ears to be bundled withother features captured in the dictionary elements because “ears” wouldhave little informational content in distinguishing the faces in thisparticular training set. Accordingly, when selecting a training set itcan be of value to have sufficient variety for a wide range of featuresso that informative features do not get excluded from the dictionaryelements by the sparsity constraint applied to A. Note that whilesparsity is effective in driving the formation of independent dictionaryelements, maximal independence is not guaranteed.

In the factorization of the training set matrix X, the goal is toproduce matrices A and Φ that minimize the error between the two sidesof the equation:

X=ΦA

while requiring that all elements in Φ and A are non-negative (i.e., Φ≥0and A≥0) and A is sparse. Sparseness can be measured in different forms.The most common measures are the L₁ and L₀ norms. When an L₁ measure ofsparseness is used, then the sum of the absolute values of thecomponents of the pseudo-image will be minimized, whereas if an L₀measure of sparseness is used, then the pseudo-images will have aminimized number of elements, i.e., a minimized number of positiveelements because of the non-negativity constraint. When sparseness isdefined by its L₁ norm, the minimization problem takes the form of:

${{\underset{A,\Phi}{argmin}\frac{1}{2}{{X - {\Phi A}}}_{2}^{2}} + {\lambda{A}_{1}}},{{{subject}\mspace{14mu}{to}\mspace{14mu} A} \geq {0\text{;}\mspace{14mu}\Phi} \geq 0}$

Here, ∥⋅∥_(p) denotes a L_(p) norm, i.e. the p-th root of the sum ofp-th powers of absolute values (p>0). In this notation, ∥⋅∥₁ denotes theL₁ norm of a vector a or a matrix A, i.e., the sum of the absolutevalues of all coefficient values in a or A. Thus, the process to solvethis problem requires the minimization of the Frobenius norm difference(i.e., the Euclidean distance) between the two sides of the equation andthe minimization of the L₁ norm.

For the L₀ norm, which is the number of non-zero elements, theminimization problem takes for the form of:

${{\underset{A,\Phi}{argmin}\frac{1}{2}{{X - {\Phi A}}}_{2}^{2}} + {\lambda{A}_{0}}},{{{subject}\mspace{14mu}{to}\mspace{14mu} A} \geq {0\text{;}\mspace{14mu}\Phi} \geq 0}$

Note that the L₀ norm is not a classical norm definition. Also, the L₀norm is not generally used in practice since L₀ minimization is an NPhard problem. Using the L₁ norm not only provides a measure of sparsityon its own, but also provides the closest convex surrogate to the L₀norm, when solving the minimization problem. It is also possible todefine sparseness using an L_(p) norm where p is greater than zero andless than one, and usually small (e.g., p=10⁻⁵). In addition to L₀, L₁,and L_(p) (0<p<1), the sparseness measure can take other forms, such as,the one defined by Hoyer (Hoyer 2004) or the one referred to as the GiniIndex (Hurley and Rickard 2009). Note that in the above expressions, λis a parameter used to tune the strictness of the sparseness constraint.In practice, the value of λ can be selected by the algorithm as theprocess proceeds. A representative, but not limiting, example of asuitable algorithm for selecting λ as a function of iteration number isset forth below.

In practice, the process to perform sparse, non-negative BSS is a convexoptimization problem. A general outline of a suitable algorithm is setforth below in Algorithm 1, which first initializes Φ and A to benon-negative random matrices to seed the computation, and then iteratesthe computation process to satisfy the constraints imposed untilconvergence (defined for this particular algorithm by a lack of netmovement of the function's gradient) is achieved. In the examples,specifically, in the generation of the Φ matrices used in the examples,the non-negative blind source separation algorithm nGMCA (Rapin, Bobinet al. 2013, Rapin, Bobin et al. 2013) was used. This BSS algorithm is aspecific example of Algorithm 1. Sparsity was measured using the L₁norm. At each iteration i, the value of A for the last iteration(A_(i-1)) was used as the initial value for determining (A_(i)), andlikewise, the value of Φ for the last iteration (Φ_(i-1)) was used asthe initial value for determining (Φ_(i)).

  Algorithm  1 : Initialize  Φ₀  and  A₀  with  non-invasive  random  numbers:  initialize  I;  For  i = 1,  I  Normalize  the  columns  of  Φ_(i − 1)to  unit  length ;$\mspace{20mu}{\left. A_{i}\leftarrow{{\underset{A}{argmin}\frac{1}{2}{{X - {\Phi_{i - 1}A}}}_{2}^{2}} + {\lambda_{i}{A}_{1}}} \right.,\mspace{20mu}{A \geq 0}}$$\mspace{20mu}{\left. \Phi_{i}\leftarrow{\underset{\Phi}{argmin}\frac{1}{2}{{X - {\Phi\; A_{i}}}}_{2}^{2}} \right.,\mspace{20mu}{\Phi \geq 0}}$  End  For  Loop  when  gradient  descent  stops  or  if  i > I  Output  Φ_(i)  and  A_(i)

The value of lambda in this algorithm varies with the iteration numberi. Typically, lambda begins with a large value to force a high level ofsparsity at the beginning of the process and then decreases with higheriterations, the final value typically being less than or equal to 1.0For example, lambda can be calculated from formulas of the followingtype, it being understood that other formulas can be used if desired:

λ₁ = Φ₀^(T)(Φ₀A₀ − X)_(∞) λ_(i + 1) = λ_(i) − θ(λ_(i) − σ_(res))$\theta = \frac{1}{\left( {{0.8*I} - i} \right)}$

In this formula, i is the iteration number and σ_(res) is the standarddeviation of the elements of X−Φ_(i)A_(i) where the elements are treatedas a set of numbers.

With regard to initializing I, i.e., the maximum number of iterations,in the examples set forth below, I was set at 500. Skilled persons candetermine a suitable value for I for any particular application ofAlgorithm 1 or other algorithms that may be used to obtain Φ byperforming preliminary calculations using the training set of images. Inthe examples, when I was reached or the gradient descent stopped, the L₂differences between the columns of X and the columns of ΦA werecalculated as an error measure and the median of those differences wasused as an error threshold (ε) in Algorithm 2 below.

Algorithms for Generating a Pseudo-Image from a First-Image

The process of generating a pseudo-image for a first-image is a processof minimization based on the predetermined transformation matrix Φ. Itis formulated as the solution to the following problem x=Φa with thevector x being an M-dimensional vector representing the first-image andthe vector a being a K-dimensional vector constituting the pseudo-imagefor the first-image. The goal is to find the sparsest K-dimensionalvector a while maintaining a minimum error between the two sides of thex=Φa equation.

As discussed in, for example, the above linear algebra textbooks (Gill,Murray et al. 1991, Strang 2006), a necessary, but not sufficient,condition for a unique solution to this problem is that M≥K. If M>K,there is either a unique solution or no solution; a unique solutionexists if M=K and Φ is full rank; no unique solution exists if M=K and Φis not full rank; no unique solution exists if M<K.

When K (the number of element weights in the pseudo-image for thefirst-image) is chosen to be larger than M(the number of components inthe first-image), the system is underdetermined and does not have aunique solution using classical linear algebra methods. Nevertheless,because of the sparseness and non-negativity constraints, surprisingly,the system achieves effective image recognition.

In the disclosed method, a key property of the pseudo-images producedduring the production of the predetermined transformation matrix is thatthey are sparse, meaning that in a given pseudo-image for a first-imageof the training set, only a small fraction of the elements (e.g., lessthan or equal to 20%, or less than or equal to 10%, or less than orequal to 5%, or less than or equal to 1%) is active (i.e., substantiallygreater than zero, e.g., greater than or equal to 1% or greater than orequal to 5% or greater than 10% of the largest element weight). Withthis property, theories developed independently by Donoho (Chen, Donohoet al. 2001, Donoho and Elad 2003, Donoho 2006, Donoho, Tsaig et al.2012), and by Candes and Tao (Candes and Tao 2005, Candes, Romberg etal. 2006, Candes, Romberg et al. 2006) show that a unique solution canbe obtained by imposing a sparseness constraint when solving theminimization problem. Whereas the sparsity measure can take differentforms, as discussed above, the most commonly used sparsity definitionsare L₀ and L₁.

An example of the process using L₁ minimization (Donoho 2006) is tosolve:

${\min\limits_{a}{{a}_{1}\mspace{14mu}{subject}\mspace{14mu}{to}\mspace{14mu}{{x - {\Phi\; a}}}_{2}}} \leq \epsilon$

where ε is an error measure of the difference between x and Φa.

The L₁-minimization problem can be implemented by a convex optimizationprocedure, for example based on the simplex method. These techniques canbe found in various books and research publications (Gill, Murray et al.1991, Dantzig and Thapa 1997, Chen, Donoho et al. 2001, Boyd andVandenberghe 2004, Candes and Tao 2005, Donoho 2006, Donoho, Tsaig etal. 2012).

Notably, the methods disclosed here have a non-negative constraint,which requires all coefficients (element weights) of the vector a to benon-negative. Thus, the problem is properly written as:

${\min\limits_{a}{{a}_{1}\mspace{14mu}{subject}\mspace{14mu}{to}\mspace{14mu}{{x - {\Phi\; a}}}_{2}}} \geq 0$

where the terminology a≥0 means that all components of a are zero orpositive.

A representative non-limiting example of an algorithm that can be usedto obtain a pseudo-image represented by the vector a using thepredetermined transformation matrix Φ is the “l₁ MAGIC” technique ofCandes and Romberg, 2005. In the examples below, Candes' and Romberg'sMin-l₁ approach with quadratic constraints was used with themodifications that the matrices were not required to be positivedefinite when obtaining inverses and the coefficient values of thevector a were required to be positive, which was achieved by settingnegative coefficients to zero at the end of the algorithm. The ϵ valuefrom Algorithm 1 was used as the error measure.

The structure of the algorithm used in the examples was:

The values of the τ₁, μ, η parameters used in the examples were asfollows:

$\tau_{1} = {\max\left( {\frac{{2m} + 1}{{a_{0}}_{1}},1} \right)}$μ = 10 η = 0.001

The goal of the algorithm is to minimize the sparsity of a vector a thatsatisfies the constraint ∥x−Φa∥₂≤ϵ. This is an optimization problemwhich dictates that we remain in the constrained region while solvingthe problem. This means that a_(i) needs to satisfy ∥x−Φa_(i)∥₂≤ϵ orϵ²−∥x−Φa_(i)∥₂ ²≥0. However, it should be noted that because the goal issparsity, merely satisfying ∥x−Φa_(i)∥₂≤ϵ is not indicative of havingarrived at a a_(i) with optimized sparsity. Rather, the end ofoptimization is reached when a parameter referred to as the duality gap

$\left( \frac{m}{\tau_{i}} \right)$

is less than a predetermined value (0.001 in the examples below).However, a direct comparison to the duality gap is not made in thealgorithm. Instead a number of iterations (I) that serves as aconvergence guarantee is calculated using the duality gap parameter, andthe algorithm is then run for those many iterations.

A cost function ƒ can be used to move a_(i) towards the desiredsolution. An example of a suitable cost function is the followingfunction, which was used in the examples:

${{def}\text{:}\mspace{14mu}{f\left( a_{i} \right)}} = {{a_{i}}_{1} + {\frac{1}{\tau_{i}}\left( {- {\log\left( {\epsilon^{2} - {{x - {\Phi\; a_{i}}}}_{2}^{2}} \right)}} \right.}}$

To minimize this cost function and thus find the desired vector a whichsatisfies ∥x−Φa∥₂≤ϵ and is sparse, the following steps can be used:

-   -   (1) calculate the number of steps required to minimize the cost        function using the duality gap parameter;    -   (2) choose a starting point a₀ that is feasible, i.e., a        starting point that satisfies the ∥x−Φa∥₂≤ϵ constraint (note        that although not used in the examples, if desired, a₀ can be a        random starting point in the feasible region);    -   (3) to reach the minimum of the cost function from the starting        point one needs to move in a direction where the value of the        function is less than the value at the starting point; to find        that direction, the cost function at the starting point is        approximated with a parabola (second order approximation);    -   (4) the minimum of the parabola is then found analytically and        a₀ is maximally moved in that direction while staying in the        feasible region;    -   (5) when doing step (4), the decrease in the cost function is        checked to determine if it is within a preselected percentage,        e.g., 1.0 percent in the examples, of the decrease predicted by        a linear model of the cost function at the starting point;    -   (6) if the decrease is not within the preselected percentage,        the step size is decreased until the decrease in the cost        function comes within the preselected percentage range;    -   (7) the new point resulting from step (4) is then used as the        starting point, and steps (3) to (6) are repeated until the        slope of the cost function is below a preselected level, e.g.,        0.001 in the examples;    -   (8) once the slope is below the preselected level, the value of        τ_(i) is changed, e.g., multiplied by 10, and steps (3) to (7)        are repeated.

It should be noted that it may take more than one step to reach theminimum of the parabola. In the examples, a maximum of 50 steps wasused, i.e., either the minimum of the parabola was reached prior to 50steps, or the point reached at 50 steps was taken as the minimum.

The above structure for Algorithm 2 used a “for loop”; the followingstructure uses a “while loop” with the values of the τ₁, μ, η parametersbeing the same as above. Numerous other approaches for programming theabove procedure for obtaining a, as well as other procedures for findingsparse, non-negative vectors that satisfy the x=Φa equation, will beevident to skilled persons from the present disclosure.

In the typical case, the first-image is a grayscale image whose pixelshave numerous values. In some cases, the first-image can be a binaryimage whose pixels can only have one of two possible values (e.g., on oroff). In this case, K can be selected to be less than M so that, asdiscussed in the above linear algebra textbooks (Gill, Murray et al.1991, Strang 2006), a unique solution exists. When applied to a binaryimage having K less than M, the above techniques find that uniquesolution. However, although sparseness is applied, the unique solutionturns out to be not particularly sparse. First-images of symbolstypically fall within this category where unique solutions are possible.

It should be noted that the disclosed methods are different from thoseused in compressed sensing and sparse signal recovery (Donoho 2006, Elad2010, Eldar and Kutyniok 2012) because in these methods, the goal was toreconstruct or approximate the original signal fiducially. In thedisclosed method, the created pseudo-image bears no resemblance to theoriginal-image and is created in different dimensions. Using a sparse,non-negative transformation, pseudo-images generated from variations ofthe first-image, as well as corrupted or occluded first-images, can benearly identical to the pseudo-image of the unadulterated first-image,as shown in the examples.

FIGS. 42-44 set forth representative flowcharts that can be used in thepractice of the present disclosure. These flowcharts as well as those ofFIGS. 45-46 discussed below are, of course, merely provided for purposesof illustrating embodiments of the disclosure and are not intended tolimit the scope of the invention as defined by the claims in any manner.

FIG. 42 sets forth a flowchart that can be used in constructing apredetermined transformation matrix for use in transforming first-imagesinto pseudo-images. The flowchart includes the steps of: (1) obtaining aset of facial images and, if needed, preprocessing the facial imagesinto first-images of a specified dimension M=m₁×m₂, (2) organizing thefirst-images into an X matrix, and (3) performing a sparse, non-negativefactorization of the X matrix to obtain the predetermined transformationmatrix Φ and a matrix A of pseudo-images for the training set.Thereafter, if desired, the columns of A can be organized as apseudo-image library. Also, if desired, statistical analyses, such as,PCA, hierarchal clustering, and/or analyses with support vectormachines, can be performed on the matrix A in order to classify thepseudo-images. As just one example, using techniques of this type, humanfaces can be classified as male or female faces.

FIG. 43 sets forth a flowchart that can be used in performing imagerecognition, specifically, facial recognition. The flowchart includesthe steps of: (1) obtaining a facial image of interest (anoriginal-image-of-interest), (2) if needed, preprocessing theoriginal-image into a first-image-of-interest of a specified dimensionM=m₁×m₂, (3) using a predetermined transformation matrix to perform asparse, non-negative transformation of the first-image-of-interest toproduce a pseudo-image-of-interest, (4) comparing thepseudo-image-of-interest with at least one library of pseudo-images, and(5) reporting the results of the comparison.

FIG. 44 sets forth a flowchart that can be used in preparing oraugmenting one or more pseudo-image libraries. The flowchart includesthe steps of: (1) obtaining a facial image to be included in the one ormore pseudo-image libraries, (2) if needed, preprocessing the facialimage into a first-image of a specified dimension M=m₁×m₂, (3) using apredetermined transformation matrix to perform a sparse, non-negativetransformation of the first-image to produce a pseudo-image, and (4)incorporating the pseudo-image along with at least someindexing/identification information into the one or more pseudo-imagelibraries.

The steps set forth in the flowcharts of FIGS. 42-44 discussed above andFIGS. 45-46 discussed below or in other flowcharts developed based onthe present disclosure can be readily implemented using a variety ofcomputer equipment and a variety of software programming languages,e.g., MATLAB or OCTAVE, which are well-suited for matrix calculations.Other programming languages that can be used in the practice of thedisclosure include, without limitation, FORTRAN, C, C++, PYTHON, PASCAL,BASIC, and the like. More than one programming language can be used inthe practice of the disclosure if desired.

Output from the computations can be in electronic and/or hard copy form,and can be displayed in a variety of formats, including in tabular andgraphical form. For example, graphs can be prepared using commerciallyavailable data presentation software such as those that are part ofMATLAB and OCTAVE or those of MICROSOFT's EXCEL program, R, or othersoftware packages.

Programs for implementing the disclosure can be provided to users on anon-transitory, computer-readable medium with instructions storedthereon capable of being executed by a computer processor to perform thesteps of the process. Non-limiting examples of such media includediskettes, CDs, flash drives, and the like. The programs can also bedownloaded to users through the internet. In addition, the process ofthe disclosure can be provided to users on-line through, for example,“cloud” computing. The process can be performed on various computingplatforms, including personal computers, workstations, mainframes,supercomputers, etc.

The predetermined transformation matrix can be implemented as computerhardware, including computer hardware that is field programmable. Forexample, the predetermined transformation matrix can be directlyprogrammed into a computer chip, e.g., a microchip, and can be alterablein the field through the use of a programmable device, e.g., a FPGA.Once a large enough training set has been employed in determining apredetermined transformation matrix, no addition learning will normallybe necessary and a fixed predetermined transformation matrix can be usedacross different platforms (different machines) and provided as ahard-implemented device, e.g., as firmware. Hardware implementations maybe particularly well-suited for established image recognition systems.

As noted immediately above, once determined by the methods discussedabove, a predetermined transformation matrix will generally not requireadditional learning. However, in cases where additional learning isdesired, e.g., to improve robustness or accuracy, two approaches can beused to produce an “augmented” predetermined transformation matrix. Thetwo approaches will be referred to as the “de novo” and “sequential”approaches. Since an augmented predetermined transformation matrix thatresults from additional learning can be used in the same manner as anexisting predetermined transformation matrix when performing imagerecognition, the term “predetermined transformation matrix” will beunderstood to include both an existing (e.g., original) predeterminedtransformation matrix and an augmented predetermined transformationmatrix produced by additional learning. It will also be understood that,if desired, augmentation can be performed multiple times using eitherthe de novo approach, the sequential approach, or a combination of thoseapproaches.

The de novo approach to additional learning uses the methods discussedabove for producing an original predetermined transformation matrix toproduce the augmented predetermined transformation matrix. In accordancewith the methods discussed above, all of the training images areutilized at once by forming the M×N dimension X matrix of first-imagesand then factorizing the X matrix into the M×K dimension predeterminedtransformation matrix (the Φ matrix) and the K×N dimension A matrix ofpseudo-images that correspond to the first-images of the training set.

In accordance with the de novo approach, this process of using all ofthe first-images of the training set at once is repeated but with alarger (augmented) training set, i.e., an X matrix with more columns.Specifically, an Mx (N+N′) dimension X matrix is formed where N′ is thenumber of newly-incorporated images (N′≥1). This matrix is then factoredinto an M×K dimension predetermined transformation matrix (the augmentedΦ matrix) and a K×N+N′ dimension A matrix of pseudo-images thatcorrespond to the first-images of the augmented training set. Becausethe process involves an initial seeding of the A matrix (and the Φmatrix) with non-negative random numbers (see Algorithm 1 above), theresulting augmented predetermined transformation matrix will, ingeneral, be sufficiently different from the existing (prior)transformation matrix so as to require recomputation of pseudo-imagelibraries that were generated with the prior matrix. Thus, in additionto being time consuming, this approach may disrupt previously designatedassociations between the pseudo-images and other datasets (e.g.,criminal records, etc.).

In accordance with the second approach—the sequential approach—the needfor such recomputation can be substantially reduced or eliminatedcompletely. As its name implies, the sequential approach performssequential learning in which the Φ matrix and the A matrix are updatedbased on one or more new first-images incorporated into the training setwithout the need to start over from the beginning as in the de novoapproach. The method offers an advantage over de novo learning of beingmore efficient. Importantly, in general, it is able to update thepseudo-images for the training set without affecting their identities.

As a preliminary step before using either the de novo or the sequentialapproaches to augment an existing predetermined transformation matrix,it will normally be appropriate to determine whether that matrixactually needs augmentation in order to be able to generatepseudo-images for a new first-image or a set of new first-images. FIG.45 sets forth an exemplary flowchart for performing such a preliminaryinquiry. As shown in the first box of that figure, the inputs to theprocess are the existing predetermined transformation matrix Φ₀ and thenew first-image set Y, which may be a single new first-image. Because,as discussed below, the sequential approach uses the existingpseudo-image set A₀ corresponding to the training set used to produceΦ₀, A₀ is also shown as an input in FIG. 45, although it is not usedwhen the de novo approach is employed.

As shown in the calculation step of FIG. 45 (the second box of thatfigure), using Φ₀ and Y, a set of pseudo-images Â is calculated for Yusing procedures analogous to those of Algorithm 1 above, but with aminimization only over A as opposed to over both A and Φ. Thus, λ and Iare as in Algorithm 1, as is ε used in the decision box (yes/no box) ofFIG. 45. As shown in the decision box, when the error E₀ is less than orequal to e, Φ₀ can continue to be used without augmentation. The processalso generates the pseudo-images Â for the one or more new first-imageswhich, for example, can be used to augment one or more pseudo-imagelibraries. Thus, while Φ is not augmented, the pseudo-images areaugmented through the calculation of Â.

When the calculation box of FIG. 45 produces an E₀ value that is greaterthan ε, the process proceeds to FIG. 46. (Note that, if desired, theprocess of FIG. 46 can be performed without first performing the processof FIG. 45; likewise, the de novo approach can be commenced withoutfirst performing the process of FIG. 45.) The process of FIG. 46 assumesthat a set of N first-images have been used to produce a predeterminedtransformation matrix Φ₀ and a corresponding set of pseudo-images A₀.When a new first-image or a set of new first-images Y is to beincorporated in the system, the method searches for a new predeterminedtransformation matrix Φ (the augmented predetermined transformationmatrix) and pseudo-images A that, using L₁ minimization, minimize thecost function:

${{\underset{\Phi,A}{argmin}\frac{1}{2}{{\left\lbrack {Y,{\Phi_{0}A_{0}}} \right\rbrack - {\Phi A}}}_{2}^{2}} + {\lambda{A}_{1}}},{\Phi \geq 0},{A \geq 0}$

A general outline of the process for performing this minimization is setforth in FIG. 46 and a representative non-limiting example of analgorithm that can be used with this general outline is set forth belowin Algorithm 3. As will be evident to skilled persons from the presentdisclosure, other algorithms and general outlines for performing thesequential process can be used if desired.

Algorithm 3: Concatenate matrix of new images with Φ₀A₀, which are theproxy of the first-images of the previous training set. Initialize Φ₁ =Φ₀; initialize A′ ∈ (K, N′) with random numbers; initialize A₁ = [A′,A₀]; initialize I; For i = 2, I    Normalize the columns of Φ_(i−1) tounit length;   $\quad\begin{matrix}{\left. A_{i}\leftarrow{{\arg\;{\min\limits_{A}{\frac{1}{2}{{\left\lbrack {Y,{\Phi_{0}A_{0}}} \right\rbrack - {\Phi_{i - 1}A}}}_{2}^{2}}}} + {\lambda{A}_{1}}} \right.,{A \geq 0}} \\{\left. \Phi_{i}\leftarrow{\underset{\Phi}{\arg\;\min}\frac{1}{2}{{\left\lbrack {Y,{\Phi_{0}A_{0}}} \right\rbrack - {\Phi\; A_{i}}}}_{2}^{2}} \right.,{\Phi \geq 0}}\end{matrix}$ End For Loop when gradient descent stops or if i > IOutput Φ_(i) and A_(i)

The values of I and λ used in Algorithm 3 are determined in the samemanner as discussed above in connection with Algorithm 1. As inAlgorithm 1, at each iteration i, the value of A for the last iteration(A_(i-1)) is used as the initial value for determining (A_(i)), andlikewise, the value of Φ for the last iteration (Φ_(i-1)) is used as theinitial value for determining (Φ_(i)).

The process of FIG. 46 and Algorithm 3 is able to substantially preserveexisting pseudo-images for the prior members of the training set as wellas existing pseudo-image libraries. This is especially so when theexisting predetermined transformation matrix Φ₀ is robust. Anexamination of Algorithm 3 reveals that rather than concatenating Y withthe original training set of first-images (the X matrix), Y isconcatenated with Φ₀A₀ as a proxy for X. Thus, Φ₀ is actively used inthe process and can thereby influence the contents of the augmentedpredetermined transformation matrix and thus the pseudo-images producedusing that augmented matrix. When Φ₀ is robust, the augmentedpredetermined transformation matrix need not be excessively differentfrom Φ₀ to cover the new first-images being introduced into the trainingset in the augmentation process. Consequently, pseudo-images generatedwith the original predetermined transformation matrix (Φ₀) and theaugmented predetermined transformation matrix need not be excessivelydifferent thus making it more likely that previously-designatedassociations between the existing pseudo-images and other datasets canbe preserved.

FIG. 47 schematically illustrates a non-limiting architecture ofcomponents of a computer system 200 for performing image recognitionusing the methods disclosed herein. In this non-limiting, exemplaryembodiment, system 200 includes one or more computer processors 201 andone or more memories 203 with data and instructions stored therein that,when used by the one or more computer processors, can perform the stepsof transforming a first-image into a pseudo-image and then comparing thepseudo-image with a library of pseudo-images and/or incorporating thepseudo-image in one or more pseudo-image libraries. The one or morecomputer processors and one or more memories can also be used totransform original-images into first-images. The same or a separatecomputer system can be used to calculate one or more predeterminedtransformation matrices for use in performing transformations fromfirst-images to pseudo-images. A predetermined transformation matrix 205and a pseudo-image library 207 are shown separately in FIG. 47, it beingunderstood that they can be part of one or more memories 203 or can behard coded into the one or more computer processors 201.

In addition to its processor/memory unit 209, computer system 200 canalso include an I/O device 211 that transmits acquired signals throughan I/O interface 213 to the processor/memory unit. These I/O devices canbe used to, for example, load original-images, first-images,pseudo-image libraries, and/or predetermined transformation matricesinto the system's memory. The devices can also be used to transmitoperator commands to the system. The results of the computationsperformed by processor/memory unit 209, e.g., pseudo-images,predetermined transformation matrices, pseudo-image libraries,comparison reports, and the like, can be output through output/displayunit 215 and/or stored in a non-transitory, computer-readable storagemedium 217.

Without intending to limit its scope in any manner, the invention isfurther illustrated by the following non-limiting examples.

Example 1

This example illustrates the application of the image recognitiontechniques disclosed herein to the problem of facial recognition.

FIG. 1 shows an original-image of a person-of-interest whose identify isdesired. The person-of-interest is in a crowd of other people as willoften be the case in applications of the facial recognition techniquesdisclosed herein. FIG. 2A shows an initial (first round) of first-levelpreprocessing in which FIG. 1 was cropped to highlight just the face ofthe person-of-interest. The dimension of the cropped face had the samenumber of vertical and horizontal pixels; specifically, the croppedimage had 10,000 pixels (100×100). The cropping was performed using theopen-source software OPEN-CV, but could also have been done using facedetection software, such as, GOOGLE VISION API or CLANDMARK.

FIG. 2B shows further first-level preprocessing in which the FIG. 2Aimage was down-sampled to a pre-specified dimension (m₁×m₂=25×25). Inthis example, down-sampling was achieved by local averaging over 4×4pixel subareas. This down-sampling reduced the 100×100 pixels of FIG. 2Ato 25×25 pixels for FIG. 2B. The grayscale values for the pixels werenormalized to be between zero and one. The FIG. 2B image was thefirst-image-of-interest (step(a)-first-image) for this example and thusM for this example was 625.

A training set of 2,000 facial images (N=2,000) was obtained from anautomated web-based image search and subjected to the same first-levelpreprocessing used to obtain FIG. 2B from FIG. 2A. FIG. 3 shows five ofthe facial images of the training set.

Using the full set of 2,000 training images and Algorithm 1 above, a Φmatrix, i.e., a predetermined transformation matrix, was obtainedcomposed of 1,500 dictionary elements (i.e., K=1,500 and Φ was a625×1,500 matrix). FIG. 4 shows 36 of the 1,500 dictionary elementsobtained in this way. In FIG. 4, the dictionary elements are displayedas two-dimensional arrays, rather than as columns of the predeterminedtransformation matrix.

As discussed above, the process of producing the predeterminedtransformation matrix Φ generates a pseudo-image for every image in thetraining set. Each pseudo-image has the same number of element weights(components) as the number of columns (number of dictionary elements) inΦ, i.e., each pseudo-image has K element weights, which in this examplewas 1,500. FIG. 5 shows five of the 2,000 pseudo-images generated inthis way. In this figure, the element weights of the pseudo-images aredisplayed as two-dimensional, grayscale arrays, rather than as vectorsof numerical values as would be their typical form in a computer system.As can be seen, objects from the real world (i.e., human faces) can beperceived in the first-images but not in the pseudo-images.

FIG. 6 shows the active elements of the pseudo-image corresponding tothe face shown in FIG. 2B, where active elements are dictionary elementshaving coefficients (element weights) substantially greater than zero.The pseudo-image of this figure was obtained using Algorithm 2 and thepredetermined transformation matrix some of whose dictionary elementsare shown in FIG. 4.

The top 12 dictionary elements of the pseudo-image, i.e., the 12dictionary elements with the largest element weights, are marked by darkframes in the bottom panel of FIG. 6, which shows all 1,500 dictionaryelements. The grayscale of each element in the pseudo-image indicatesthe coefficient value of that element.

An alternative view of the active elements, including values for thecoefficients (element weights) of the significant dictionary elements,is shown in FIG. 7. The coefficient value of each element is indicatedby the height of the line corresponding to the element.

The identification of the face shown in FIG. 2B and thus in FIG. 1 wasperformed by calculating a similarity score between thepseudo-image-of-interest, i.e., the pseudo-image of FIG. 6, and eachmember of a library of pseudo-images. For the purposes of this example,the pseudo-image library was the set of pseudo-images for the trainingset which, as discussed above, were generated at the same time thepredetermined transformation matrix Φ was generated. Accordingly, the Svalue for the pseudo-image library was 2,000. The function cos(θ) wasused as the similarity score for this example, as well as in Examples2-10.

FIG. 8 shows the reporting of the identification of thefirst-image-of-interest and thus the original-image by a comparison ofthe pseudo-image-of-interest of FIG. 6 with the full pseudo-imagelibrary from which the portion of FIG. 5 was taken. The first-image withthe highest score is identical to the first-image-of-interest. As can beseen, its similarity score is substantially greater than the nexthighest score, thus illustrating the robustness of the identificationfor this system where K/M was 2.4 (K=1,500; M=625). The rank of the Xmatrix, as determined by the RANK( ) function in MATLAB, was 625 so thatthe K/R and K/M values for this example were both 2.4, each of which isindicative of robustness. The K/M and K/R value for this Example 1 wasalso the K/M and K/R value for Examples 3-6 below.

Example 2

This example illustrates how reducing the values of the K/M and K/Rratios compromises the robustness of the image recognition procedure.

The same procedures and training set as in Example 1 were used with theK value set at 500 instead of 1,500 and thus, although the pseudo-imagelibrary still had 2,000 pseudo-images, the pseudo-images were differentbecause K was different. The K/M and K/R values were both 0.8 since theX matrix was the same as in Example 1 and thus had the same R equal to Mvalue as in that example, i.e., 625. The 0.8 value for K/M and K/R forthis Example 2 was also the K/M and K/R value for Example 7 below.

FIGS. 9-13 show the results. Comparing FIG. 9 with FIG. 4 of Example 1,we see that reducing K changed the dictionary elements. For the smallerK value, the dictionary elements incorporated less features and weresketcher than those in FIG. 4, but still resembled faces.

Comparing FIGS. 10-12 with FIGS. 5-7 of Example 1, we see that reducingthe K value resulted in more active elements, meaning that thepseudo-images were less sparse. The reduction of sparseness has animpact on the comparison of the pseudo-image-of-interest with thelibrary of pseudo-images. Specifically, as shown in FIG. 13, many morefaces in the library of pseudo-images have moderate values in thesimilarity score, as compared with the few in FIG. 8. Although the samecorrect face was identified, reducing the K value caused the difference(contrast) between the similarity score for the correct face and thatfor the nearest runner up to be much smaller for this example than forExample 1. Hence, the robustness was compromised.

This reduction in robustness is further demonstrated below by Examples3-6, where imperfect facial images of various types were successfullyidentified when K was 1,500, and Example 7, where successfulidentification was not achieved when K was reduced to 500, thus takingthe K/M and K/R ratios from above 1.0 for Examples 3-6 to below 1.0 forExample 7.

Example 3

This example illustrates the ability of the technique to perform facialrecognition for people having part of their face obscured by, in thiscase, glasses or sunglasses. The same procedures, training set, andpseudo-image library as in Example 1 were used except that for thisexample, the first-images-of-interest were faces from the training setmodified by including a pair of glasses or sunglasses around the eyes.The faces in the training set remained the same; only thefirst-images-of-interest whose identities were sought were changed.

FIGS. 14 and 15 show the results. The pseudo-images obtained using thepredetermined transformation matrix Φ of Example 1 and Algorithm 2showed more active elements, specifically, for FIGS. 14 and 15,respectively, 4.3% and 2.6% active elements for the original first-imagefor a threshold of 0.3 percent of the largest element weight versus18.9% and 19.1% for the modified (imperfect) first-image using thethreshold used for the original image, i.e., the threshold based on thelargest element weight for the original first-image. However, thecoefficient values for the additional elements were relatively small,i.e., substantially invisible in the middle columns of FIGS. 14 and 15.Thus, although the sparseness of the pseudo-images for the imperfectfirst-images was compromised, there still was substantial sparseness.Importantly, as can be seen in these middle columns, the main, highelement weight (high coefficient value) dictionary elements are the samebetween the modified facial image and the original one.

As shown in the right hand columns of FIGS. 14 and 15, the similarityscores clearly identify the original face as the correct onenotwithstanding the fact that the first-images-of-interest used in theanalysis were wearing the eyewear.

Example 4

This example illustrates the ability of the technique to perform facialrecognition for people with facial expressions that are not in thetraining set. The same procedures, training set, and pseudo-imagelibrary as in Example 1 were used. A face in the training set wasmodified by changing from smiling to non-smiling (top panel of FIG. 16)or from non-smiling to smiling (bottom panel of FIG. 16). The modifiedfaces were then used as first-images-of-interest, the images in thetraining set being unchanged, i.e., the predetermined transformationmatrix Φ and the library of pseudo-images used for the comparison wereunchanged.

The right hand column of FIG. 16 shows the similarity scores between thefaces with the different expressions and the pseudo-images of thelibrary. The highest scores correctly identified the original faces.

Example 5

This example illustrates the ability of the technique to perform facialrecognition for people having multiple parts of their face obscured by,in this case, sunglasses, facial hair, or both. The same procedures,training set, and pseudo-image library as in Example 1 were used. Forthis example, the first-images-of-interest were a face from the trainingset modified by including a pair of sunglasses around the eyes, amustache above the mouth, or both. The face in the training set remainedthe same; only the first-images-of-interest whose identities were soughtwere changed.

FIG. 17 shows the results. The largest similarity score between modifiedfaces and the library of pseudo-images for the training set is theoriginal face notwithstanding the fact that the first-images-of-interestused in the analysis were wearing a pair of sunglasses around the eyes,a mustache above the mouth, or both.

Example 6

This example illustrates the ability of the technique to perform facialrecognition for people having part of their face obscured by an objectwhich in practice could be a hat, a scarf, a mask, or the like. The sameprocedures, training set, and pseudo-image library as in Example 1 wereused. For this example, the first-images-of-interest were a face fromthe training set obstructed by a black mask that removed half of theface from being seen. The mask was placed to obscure different parts ofthe face (i.e., the top, bottom, left, or right halves). FIGS. 18 and 19show the similarity scores between the obscured faces and the library ofpseudo-images for the training set. In two of the cases (left hand casesin FIG. 18), the correct face was one of the faces with a highsimilarity score, although not the face with the highest score; in theother six cases (right hand cases in FIG. 18 and all the cases in FIG.19), the original face was the face with the highest score.

Example 7

This example shows the effect of reducing the K value of 1,500 that wasused in Examples 3-6 to 500. As noted above, with the reduction in K,the K/M and K/R value for this example was 0.8.

Specifically, FIGS. 20 and 21 correspond to FIGS. 14 and 15 of Example3, FIG. 22 corresponds to FIG. 16 of Example 4, FIG. 23 corresponds toFIG. 17 of Example 5, and FIGS. 24 and 25 correspond to FIGS. 18 and 19of Example 6. The same procedures and training set as in Example 1 wereused except that for this example, K was 500 instead of 1,500 and thus,although the pseudo-image library still had 2,000 pseudo-images, thepseudo-images were different because K was different.

As can be seen in FIGS. 20-25, the wrong face was identified in eachcase thus illustrating the lack of robustness in identifying imperfectimages when the K/M and K/R values were less than 1.0.

Example 8

This example illustrates the ability to achieve even greater robustnessin image recognition performed on imperfect first-images by increasingthe value of K and thus the values of K/M and K/R.

Example 5 and the female face of Example 6 were repeated with K equal to2,500 instead of 1,500, so that the K/M and K/R ratios were each 4.0instead of 2.4. FIGS. 26 and 27 show the results of the analysis.

As can be seen in these figures, the similarity score for the correctface is now much greater than the nearest runner up, i.e., the contrastbetween similarity scores is greater. Compared to FIGS. 17 and 18 whereK was 1,500, the correct face for FIGS. 26 and 27 with K equal to 2,500now stands substantially alone in terms of similarity score. Also, forthe female face of Example 6, the correct face is now identifiedirrespective of the part of the face that is obscured. It has been foundthat female faces tend to be more difficult to identify than male facesand thus may require somewhat larger K/M and/or K/R values for robustidentification as the results of this example and Example 6 illustrate.

Example 9

This example illustrates the ability of the technique to perform facialrecognition for people who are not in the training set.

The images used were from the Yale facial image database which can befound at http://cvc.cs.yale.edu/cvc/projects/yalefaces/yalefaces.html.Specifically, 15 individuals, each having 11 different facialexpressions or lighting conditions, were used. FIG. 28 shows the 165faces that were used. Each row in FIG. 28 is for a different individualwith the different facial expressions or lighting conditions for thatindividual being shown as one moves across a row.

Each facial image in FIG. 28 was treated as a first-image-of-interestand was transformed into a pseudo-image-of-interest using the sameprocedures and training set as in Example 1 but with K equal to 2,500instead of 1,500 thus making the K/M and K/R ratios 4.0 instead of 2.4.The similarly scores between the resulting pseudo-images-of-interesttaken as pairs were then calculated.

FIG. 29 shows the pairwise scores among the 15 individuals and their 11faces (a total of 27,225 comparisons). Darker grayscales indicate highersimilarity scores. Faces belong to the same person are grouped andindicated by the number along the axes. The groupings along the diagonalin this figure show that a given person has high similarity scoresbetween his/her own facial images even with different expressions andlighting conditions. On the other hand, the similarity scores betweenpseudo-images of different people are low. The graph thus shows thatdespite different facial expressions and lighting conditions, facesbelonging to the same person are highly similar in their pseudo-images,but are not similar to those belonging to a different person.

FIG. 30 shows the similarity scores between the pseudo-images for thefaces from the Yale database of FIG. 28 and the pseudo-images for thefaces of the training set of Example 1. The similarity scores betweenthe pseudo-images for these two sets of unrelated faces are low.

Thus, the technique is able to determine if a given individual is or isnot part of a pseudo-image library and is also able to groupfirst-images of a particular individual having different facialexpressions and different lighting conditions even if that individualwas not part of the training set used to generate the predeterminedtransformation matrix used in producing pseudo-images.

Example 10

This example illustrates the ability of the technique to performrecognition of symbols.

FIG. 31 shows 1,000 letters and characters from world languages. These1,000 symbols were used as a training set (i.e., N=1000) to produce apredetermined transformation matrix Φ with 800 dictionary elements(i.e., K=800), which was then used to produce pseudo-images for theEnglish capital letter “H” and a Chinese “bing” character, with andwithout adulteration.

Each of the symbols of the training set, as well as the “H” and the“bing” character used in the tests, had 256 pixels (i.e., M=256), givinga K/M ratio of 3.1 for the Φ matrix. The predetermined transformationmatrix thus transformed each symbol from a 256-dimensional space to an800-dimensional space. The rank of the X matrix as determined using theMATLAB RANK( ) function referred to above in Example 1 was 253, so theK/R ratio was 3.2. The A matrix produced during the calculation of Φ wasused as the pseudo-image library (S=1,000) for comparison with thepseudo-images of the “H” letter and the “bing” character, both in theiradulterated and unadulterated forms. As in the prior examples, Algorithm1 was used to generate the Φ matrix and Algorithm 2 was used intransforming first-images into pseudo-images.

FIG. 32 shows the results for the unadulterated letter “H” (top panel)and unadulterated “bing” character (lower panel). The similarity scoresshown in this figure are the values of the cos(θ) function for theelement weights of the pseudo-image-of-interest versus the elementweights of the pseudo-images of the pseudo-image library. As can beseen, the correct letter/character was found in each case. The secondhighest similarity score in the case of the English capital letter “H”was the Greek capital letter “eta” which appears in the third row, lastcolumn of FIG. 31. As can be seen, the differences between this Greekletter and the English letter are minimal and yet the image recognitionprocedures disclosed herein were able to distinguish between these twoletters.

FIG. 33 shows the results for corrupted first-images, specifically,first-images where pixels are missing (pixel value set equal to zero).Again, the procedure readily identified the correct letter/characternotwithstanding the corruption of the image.

As noted above, for binary images, such as the symbols of FIG. 31, therequirements on K for robust image recognition can often be relaxed.This effect is illustrated in FIG. 34 which repeats FIG. 33 but with Kequal to 100, instead of 800. Thus, instead of the K/M ratio of 3.1 andthe K/R ratio of 3.2 of FIG. 33, FIG. 34 had K/M and K/R ratios of only0.4.

As can be seen in FIG. 34, the procedure was able to identify thecorrupted letter “H” and the corrupted “bing” character even with thislow value of K. Compared to FIG. 33, the robustness was plainlycompromised as evidenced by the large number of symbols with substantialsimilarity scores, but the system was still robust enough to find thecorrect symbol.

FIG. 35 further characterizes the effect of changing K from 800 to 100.The panels of this figure plot the error, in particular, the value of1−cos(θ), between the pseudo-image-of-interest and the correctpseudo-image as a function of the number pixels in thefirst-image-of-interest. Specifically, random sets of pixels from eachof the 1,000 symbols of FIG. 31 were used as first-images-of-interest,these first-images-of-interest were converted topseudo-images-of-interest, and then those pseudo-images-of-interest werecompared with the pseudo-image for the unadulterated symbols. The numberof pixels in the random sets is plotted along the horizontal axis andthe 1−cos(θ) values along the vertical axis. Specifically, the datapoints are the means for the 1,000 symbols, the solid line is for themedians, and the shading is for the variances for the cosine error. Theupper panel is for K=800 and the lower panel is for K=100.

As can be seen in the lower panel of FIG. 35, even for K=100, thelikelihood of a correct identification is greater than ˜80% when thenumber of pixels is greater than ˜50% of the total number of pixels. ForK=800, the robustness is remarkably better, with the median correctidentification reaching 100% with only 40 out of 256 pixels (15.6%)being present in the first-image-of-interest. This result illustratesthe unexpected power (unexpected robustness) of using pseudo-images and,in particular, pseudo-images where first-images-of-interest have beentransformed into a higher dimensional space, to perform imagerecognition.

Example 11

This example compares the de novo and sequential approaches for creatingan augmented predetermined transformation matrix. As first-images, itused the 1,000 letters and characters of FIG. 36 (N=1,000), each ofwhich was a 16×16 array of binary pixels (M=256). The value of K used inthis example was 1,000, thus giving a K/M ratio of 3.9.

The M×K dimension predetermined transformation matrix was calculated intwo ways. First, all of the letters and characters of FIG. 36 were usedat one time, as would be done when using the de novo approach foraugmenting a predetermined transformation matrix. Algorithm 1 was usedto calculate the predetermined transformation matrix.

Second, the letters and characters of FIG. 36 were used one afteranother as an extreme example of the sequential approach. Algorithm 1was used for the first letter/character and thereafter Algorithm 3 wasrepeatedly used (999 times) with the predetermined transformation matrixof the last calculation being used as the existing predeterminedtransformation matrix for the following calculation.

The resulting 1,000 dictionary elements for the two approaches are shownin FIGS. 37 and 38, where FIG. 37 is for the de novo approach and FIG.38 is for the sequential approach. A visual inspection reveals the highlevel of similarity between the dictionary elements of the predeterminedtransformation matrices calculated by the two approaches.

FIG. 39 quantifies the similarity between the dictionary elements ofFIGS. 37 and 38. Specifically, this figure plots cosine similaritybetween dictionary elements learned from sequential learning and from denovo learning. The heatmap indicates the pair-wise similarity scorebetween the two learning methods. High scores (darker color) indicatehigh levels of similarity. The diagonal dark line indicates nearlyidentical elements. As can be seen, the two learning approaches producednearly identical sets of dictionary elements for the first ˜850elements. The last 150 or so were more different. It is believed thatthis is because the first 850 elements likely captured all the importantfeature combinations with the last 150 or so only improving accuracy andnot being required for robustness.

FIGS. 40 and 41 illustrate further characteristics of the predeterminedtransformation matrix obtained using the sequential approach. FIG. 40 isa plot of pairwise correlations between the 256 components of thedictionary elements of the predetermined transformation matrix. Thepresence of substantial off-diagonal values indicates substantialinformation content in individual components of the dictionary elements,as is desirable for robust image recognition. FIG. 41 is a plot ofpairwise correlations between the components of the pseudo-imagesproduced using the sequential approach for the training set (FIG. 36).The lack of substantial off-diagonal values indicates that thefirst-images have substantially unique representations when transformedinto pseudo-image space, as is desirable for robust image recognition.

INDUSTRIAL APPLICABILITY

As discussed above, one of the main applications for the imagerecognition techniques and associated computer systems disclosed hereinis in human facial recognition. In connection with this application, inan embodiment, the facial recognition techniques disclosed herein can beused to identify a person using captured images from image capturingdevices, such as cameras and video recorders, and one or more databasesto retrieve relevant information. For example, in the setting of asecurity checkpoint, a person walking through the checkpoint may beidentified directly from a facial image. Alternatively, the imagerecognition techniques disclosed herein can be used to identify a personfrom a body image or a sequence of images that captures the gaitstructure of the person. As further alternatives, combinations of two ormore of facial image data, body image data, and gait data can be used toidentify a person.

In implementations of these types, a person will be identified withouthaving to produce personal identification. Such implementations willreduce the need for other forms of identification. In the case of thecriminal-justice system, facial images, body images, and/or gait imagescan be used with an existing criminal database to identify theperpetrator of a crime or to determine whether a known criminal waspresent at a specific location at a specific time. In the case ofconsumer identification, a returning customer walking into a store maybe recognized to allow a sales clerk to recommend products based on thecustomer's purchase history. In an e-commerce setting, the disclosedtechnology can allow the use of facial images, body images, gait imagesor combinations thereof as identification, thereby removing the need forother forms of identification. With the person's identify known, theperson's credit or debit account may be directly billed, thus removingthe need for cash or credit or debit cards.

In addition to facial recognition, the disclosed technology can be usedin other forms of imaging. For example, an image of an animal or otherliving object (e.g., plant, cell, organ, tissue, or virus) can betreated in the same way as a facial image to produce a pseudo-imagewhich can then be compared with a library (database) of knownpseudo-images. The images that are analyzed can be produced by medicalimaging devices, such as, MRI, fMRI, X-ray, CT, and similar devices.Images produced by microscopes, e.g., images of blood and tissuesamples, can also be used as original-images, as well as images in theform of sequences (e.g., genetic sequences) or in the form of traces(e.g., EKG and EEG traces). The results of the comparison ofpseudo-images-of-interest with a pseudo-image library can, for example,be used as part of the diagnosis of diseases and/or in medicalprocedures.

Other applications of the technology disclosed herein include use of aperson's signature, retina, fingerprints, or other biometrics, eitherseparately or in combination, for biometric recognition purposes.Assemblies of objects (e.g., collages created by artists) can be treatedthe same way as facial images. Indeed, pseudo-images can be used toauthenticate the work of an artist or to establish the authenticity ofobjects, e.g., modern or antique furniture, alleged to have beenproduced by a particular manufacturer.

The disclosed methods can be applied to military situations to providehigh confidence recognition of potential threats and to distinguishfriendly and hostile installations under highly variable conditions. Forexample, the method can be applied to identify enemy tanks underconditions such as fog, sandstorm, smoke, twilight or night, with thetanks being camouflage or partially hidden.

The disclosed methods can be used in remote sensing using, for example,images acquired through sensors that detect patterns not directlyvisible to human eyes. For example, sonar or infrared spectral imagescan be used to, for example, recognize mineral, gas, or oil deposits.

More generally, it will be apparent to those skilled in the art that thedisclosed image recognition techniques can be used in all forms ofmachine vision. For example, the disclosed methods can be applied toimages or image sequences to identify vehicles, obstacles, traffic signsand passage conditions in an autonomous robotic device, vehicle, orvessel, and inform a central decision maker (e.g., a computer) ofexisting conditions. The disclosed methods can be used for theidentification of faulty parts in mechanical, electrical, and electronicmanufacturing. For example, using pseudo-images for faulty vs. intactelectronic circuits, the disclosed methods can be used to correctly andrapidly identify defective circuits.

Not only can the techniques be used on still images, but they can beused to recognize a person, animal, object, or pattern in an imagesequence by considering the images captured in the sequence as aconcatenated image. That is, a sequence of images of anobject-of-interest can be concatenated or transformed into a new image,and that new image can be subject to transformation and analyses usingthe disclosed methods.

Pseudo-images can also be combined to construct new first-images whichcan then be transformed into new, higher-level, pseudo-images. Thismulti-layer approach can, for example, be used in artificialintelligence applications of the image recognition techniques disclosedherein. As just one example, in a quality control setting, usingpseudo-images for the parts of a finished machine, a manufacturer candetermine if all the parts have been included in a particular finishedmachine by (i) combining the pseudo-images for the parts into afirst-image, (ii) obtaining a pseudo-image for that first-image, and(iii) comparing that pseudo-image with a pseudo-image of the actualfinished machine to determine if all the parts are present.

The disclosed techniques can be used in conjunction with search enginesto facilitate learning, identify people and objects, and retrieverelevant information. For example, search engines can be used togenerate libraries of pseudo-images which can then be compared with animage captured by an image-capturing device. The search engine canrespond to queries by identifying the person or object that is thesubject of the query. In one scenario, a person may obtain an image of aplant and send the image to a search engine, which will then return theproperly identified plant and associated information. In anotherscenario, an image of a person whom the subject may wish to learn moreabout may be sent to a search engine, which will then return the desiredinformation. For example, in a social setting, the information retrievedcan be simply a quick reminder of the time and circumstances when anearlier encounter with the person occurred. In these and otherapplications, images produced by an image-capturing device associatedwith a computer (e.g., the camera of a smartphone or a cameraincorporated in a pair of glasses) can be used to search an existingdatabase (from a search engine provider or a personal database stored onthe device) in real time to retrieve desired information through themedium of a pseudo-image comparison.

FEATURES OF THE DISCLOSURE

Based on the foregoing, in addition to the six aspects of the disclosureset forth above in the Summary and General Description, the inventionincludes, but is not limited to, the following features. The six aspectsand the following features, as well as their various paragraphs andsubparagraphs, can be used in any and all combinations.

Feature 1: A method comprising:

(a) receiving an image in a computer system;

(b) using the computer system to perform a sparse, non-negativetransformation of the image into a pseudo-image using a predeterminedtransformation matrix;

(c) using the computer system to compare the pseudo-image with a libraryof pseudo-images of known images; and

(d) using the computer system to output the results of the comparison ofthe pseudo-image with the library of pseudo-images of known images;

wherein the image has M components, the pseudo-image has K components,and K is greater than or equal to M.

Feature 2: A method comprising:

(a) receiving an image in a computer system;

(b) using the computer system to perform a sparse, non-negativetransformation of the image into a pseudo-image using a predeterminedtransformation matrix;

(c) using the computer system to compare the pseudo-image with a libraryof pseudo-images of known images; and

(d) using the computer system to output the results of the comparison ofthe pseudo-image with the library of pseudo-images of known images;

wherein the image has M components each of which has only one of twopossible values.

Feature 3: The method of Feature 1 or 2 wherein the computer systemperforms the sparse, non-negative transformation using at least one L₂norm.

Feature 4: The method of Feature 1, 2, or 3 wherein the predeterminedtransformation matrix is a matrix obtained by a method comprising usinga computer system to perform a sparse, non-negative factorization of amatrix of training images.

Feature 5: The method of Feature 4 wherein the matrix of training imagesis an M×N matrix where Nis greater than or equal to M.

Feature 6: The method of Feature 4 or 5 wherein the computer systemperforms the sparse, non-negative factorization using at least oneFrobenius norm.

Feature 7: The method of any prior Feature wherein the image of step (a)is a pre-processed image.

Feature 8: The method of any prior Feature wherein the computer systemperforms the comparison of step (c) using at least one of a Euclideandistance and a cosine distance.

Feature 9: The method of any of Features 1, 3, 4, 5, 6, 7, 8, or 9wherein the image of step (a) comprises a human face.

Feature 10: A method of performing computer-implemented imagerecognition comprising:

(a) providing to one or more computer processors a first-image having Mcomponents;

(b) providing to the one or more computer processors a predeterminedtransformation matrix, wherein:

-   -   (i) the predetermined transformation matrix is an M×K matrix in        which the K columns constitute a set of K dictionary elements,        and    -   (ii) the predetermined transformation matrix is constructed by a        method comprising performing a sparse, non-negative        factorization of an M×N matrix in which the N columns constitute        a set of N training images, each training image having M        components, the sparse, non-negative factorization employing at        least one Frobenius norm;

(c) constructing, using the one or more computer processors, apseudo-image for the first-image using the predetermined transformationmatrix to perform a sparse, non-negative transformation of thefirst-image, said pseudo-image for the first-image consisting of Kelement weights, each element weight being for one of the K dictionaryelements, the sparse, non-negative transformation employing at least oneL₂ norm;

(d) comparing, using the one or more computer processors, thepseudo-image for the first-image with a library of pseudo-images ofknown images using at least one of a Euclidean distance and a cosinedistance; and

(e) outputting, using the one or more computer processors, the resultsof the comparison of the pseudo-image with the library of pseudo-imagesof known images;

wherein the M×N matrix has a rank R and K satisfies one or both of thefollowing relationships:

(i) K is greater than or equal to M; and

(ii) K is greater than or equal to R.

Feature 11: A method of preparing a predetermined transformation matrixfor use in image recognition comprising:

(a) providing a set of N training images to a computer system, eachtraining image having M components;

(b) using the computer system to produce a predetermined transformationmatrix by performing a sparse, non-negative factorization of an M×Nmatrix in which each of the N columns of the matrix constitutes one ofthe training images, said sparse, non-negative factorization employingat least one Frobenius norm; and

(c) storing the predetermined transformation matrix in a non-transitorycomputer readable medium;

wherein:

(i) the predetermined transformation matrix is a M×K matrix;

(ii) the M×N matrix has a rank R; and

(iii) K satisfies one or both of the following relationships:

-   -   (A) K is greater than or equal to M; and    -   (B) K is greater than or equal to R.

Feature 12: The method of Feature 11 wherein a set of pseudo-images forthe N training images is produced in step (b) and the method furthercomprises using the computer system to store at least some of thosepseudo-images in a non-transitory computer readable medium as at leastpart of a pseudo-image library.

Feature 13: The method of Feature 11 further comprising distributing thepredetermined transformation matrix as an article of commerce.

Feature 14: The method of Feature 1, 10, or 11 (or any Feature thatdepends therefrom) wherein K is greater than M.

Feature 15: A method for preparing a predetermined transformation matrixfor use in image recognition from a prior predetermined transformationmatrix comprising:

(a) providing a prior predetermined transformation matrix Φ₀ to acomputer system, said prior predetermined transformation matrix havingbeen obtained using a set of N training images;

(b) providing a set A₀ of pseudo-images for the N training images to thecomputer system;

(c) providing a set Y of N′ training images to the computer system,where N′ is greater than or equal to one and at least one member of saidset is a training image that is not part of the set of N trainingimages;

(d) using the computer system to produce a predetermined transformationmatrix using a concatenation of Y with the matrix product Φ₀A₀, wherethe Φ₀A₀ matrix product serves as a proxy for the set of N trainingimages; and

(e) storing the predetermined transformation matrix of step (d) in anon-transitory computer readable medium.

Feature 16: The method of Feature 15 wherein a set of pseudo-images forthe N′ training images is produced in step (d) and the method furthercomprises using the computer system to store at least some of thosepseudo-images in a non-transitory computer readable medium as at leastpart of a pseudo-image library.

Feature 17: The method of Feature 15 further comprising distributing thepredetermined transformation matrix of step (d) as an article ofcommerce.

Feature 18: A method of preparing or augmenting a library ofpseudo-images for use in image recognition comprising:

(a) providing a set of known images to a computer system;

(b) using the computer system to perform sparse, non-negativetransformations of the known images into pseudo-images using apredetermined transformation matrix; and

(c) using the computer system to store at least some of thepseudo-images in a non-transitory computer readable medium as at leastpart of a pseudo-image library.

Feature 19: The method of Feature 12, 16, or 18 further comprisingdistributing the pseudo-image library as an article of commerce.

Feature 20: A non-transitory computer readable medium having apredetermined transformation matrix prepared by the method of Feature 11or 15 stored therein.

Feature 21: A non-transitory computer readable medium having apseudo-image library prepared at least in part by the method of Feature12, 16, or 18 stored therein.

Feature 22: A non-transitory computer readable medium comprising alibrary of pseudo-images of known images for comparison with apseudo-image for an unknown image wherein the pseudo-images of knownimages are obtained by a method comprising performing sparse,non-negative transformations of the known images into pseudo-imagesusing a predetermined transformation matrix.

Feature 23: A non-transitory computer readable medium with instructionsstored therein capable of being executed by a computer processor toperform the steps of:

(a) transforming an image into a pseudo-image;

(b) comparing the pseudo-image with a library of pseudo-images of knownimages; and

(c) outputting the results of the comparison of the pseudo-image withthe library of pseudo-images of known images;

wherein the transformation of step (a) is a sparse, non-negativetransformation using a predetermined transformation matrix.

Feature 24: A computer system comprising the non-transitorycomputer-readable medium of Feature 23 and a computer processor forexecuting the instructions stored therein.

Feature 25: A system comprising:

a computer processor;

at least one computer memory (e.g., a RAM);

at least one computer storage device (e.g., a hard drive, a flash drive,and/or the cloud);

a computer interface that receives an image and stores the image in theat least one computer memory; and

a computer program capable of being executed by the computer processorto generate a pseudo-image for the received image and store thepseudo-image in the at least one computer storage device;

wherein the computer program is capable of generating the pseudo-imageby a method comprising performing a sparse, non-negative transformationof the image using a predetermined transformation matrix.

Feature 26: The system of Feature 25 wherein the computer program iscapable of comparing the pseudo-image with a library of pseudo-imagesand outputting a result of the comparison.

Feature 27: The system of Feature 25 or 26 wherein the computer programis capable of including the pseudo-image in a library of pseudo-images.

Feature 28: The system of Feature 25, 26, or 27 further comprising animage capture device capable of providing an image to the computerinterface.

A variety of modifications that do not depart from the scope and spiritof the invention will be evident to persons of ordinary skill in the artfrom the foregoing disclosure. The following claims are intended tocover the specific embodiments set forth herein as well asmodifications, variations, and equivalents of those embodiments.

REFERENCES

-   Boyd, S. P. and L. Vandenberghe (2004). Convex optimization.    Cambridge, UK; New York, Cambridge University Press.-   Candes, E, & Romberg, J. (2005). l1-magic: Recovery of sparse    signals via convex programming. URL:    www.acm.caltech.eu/l1magic/downloads/l1magic. pdf 4, 14.-   Candès, E. J., J. Romberg and T. Tao (2006). “Robust uncertainty    principles: Exact signal reconstruction from highly incomplete    frequency information.” Information Theory, IEEE Transactions on 52:    489-509.-   Candes, E. J., J. K. Romberg and T. Tao (2006). “Stable signal    recovery from incomplete and inaccurate measurements.”    Communications on pure and applied mathematics 59(8): 1207-1223.-   Candes, E. J. and T. Tao (2005). “Decoding by linear programming.”    IEEE transactions on information theory 51(12): 4203-4215.-   Chen, S. S., D. L. Donoho and M. A. Saunders (2001). “Atomic    decomposition by basis pursuit.” SIAM review 43(1): 129-159.-   Comon, P. and C. Jutten (2010). Handbook of blind source separation:    independent component analysis and applications. Amsterdam; Boston,    Elsevier.-   Dantzig, G. B. and M. N. Thapa (1997). Linear programming. New York,    Springer.-   Donoho, D. L. (2006). “Compressed sensing.” Information Theory, IEEE    Transactions on 52(4): 1289-1306.-   Donoho, D. L. (2006). “For most large underdetermined systems of    linear equations the minimal 11-norm solution is also the sparsest    solution.” Communications on pure and applied mathematics 59:    797-829.-   Donoho, D. L. and M. Elad (2003). “Optimally sparse representation    in general (nonorthogonal) dictionaries via 1 minimization.”    Proceedings of the National Academy of Sciences 100(5): 2197-2202.-   Donoho, D. L., Y. Tsaig, I. Drori and J.-L. Starck (2012). “Sparse    solution of underdetermined systems of linear equations by stagewise    orthogonal matching pursuit.” IEEE Transactions on Information    Theory 58(2): 1094-1121.-   Elad, M. (2010). Sparse and redundant representations: from theory    to applications in signal and image processing. New York, Springer.-   Eldar, Y. C. and G. Kutyniok (2012). Compressed sensing: theory and    applications. Cambridge; New York, Cambridge University Press.-   Gill, P. E., W. Murray and M. H. Wright (1991). Numerical linear    algebra and optimization. Redwood City, Calif., Addison-Wesley Pub.    Co.-   Hoyer, P. O. (2004). “Non-negative matrix factorization with    sparseness constraints.” The Journal of Machine Learning Research 5:    1457-1469.-   Hurley, N. and S. Rickard (2009). “Comparing measures of sparsity.”    Information Theory. IEEE Transactions on 55: 4723-4741.-   Rapin, J., J. Bobin, A. Larue and J.-L. Starck (2013). “Sparse and    non-negative BSS for noisy data.” Signal Processing, IEEE    Transactions on 61: 5620-5632.-   Rapin, J., J. Bobin, A. Larue and J.-L. Starck (2013). “Sparse    Regularizations and Non-negativity in BSS.” Proceedings of SPARS,    Lausanne, Switzerland: 83.-   Strang, G. (2006). Linear algebra and its applications. Belmont,    Calif., Thomson, Brooks/Cole.-   Yu, X., D. Hu and J. Xu (2014). Blind source separation: theory and    applications. Singapore, John Wiley & Sons Singapore Pte. Ltd.

What is claimed is:
 1. A method for protecting privacy when performingfacial image recognition comprising: (a) using a computer system tocompare a set of data representing a facial image of an individual to berecognized with sets of data representing facial images of individualswhose identities are known, at least one set of data for each of theknown individuals; and (b) using the computer system to output theresults of the comparison; wherein neither the set of data representinga facial image of the individual to be recognized nor any of the sets ofdata representing facial images of individuals whose identities areknown comprises a humanly-perceivable facial image.
 2. The method ofclaim 1 wherein the non-humanly-perceivable set of data representing afacial image of an individual to be recognized and thenon-humanly-perceivable sets of data representing facial images ofindividuals whose identities are known are in vector form and in step(a) the computer system performs the comparison using at least one of aEuclidean distance and a cosine distance.
 3. The method of claim 1wherein the non-humanly-perceivable set of data representing a facialimage of an individual to be recognized is obtained by a method thatcomprises transforming, using a computer system, a set of data thatcomprises a humanly-perceivable facial image of the individual to berecognized into the non-humanly-perceivable set of data.
 4. The methodof claim 3 wherein the transformation is a sparse, non-negativetransformation.
 5. The method of claim 4 wherein the transformation isperformed using a predetermined transformation matrix obtained by amethod comprising using a computer system to perform a sparse,non-negative factorization of a matrix of facial training images.
 6. Themethod of claim 1 wherein at least one of the non-humanly-perceivablesets of data representing facial images of individuals whose identitiesare known is obtained by a method comprising transforming, using acomputer system, a set of data that comprises a humanly-perceivablefacial image of an individual whose identity is known into anon-humanly-perceivable set of data.
 7. The method of claim 6 whereinthe transformation is a sparse, non-negative transformation.
 8. Themethod of claim 7 wherein the transformation is performed using apredetermined transformation matrix obtained by a method comprisingusing a computer system to perform a sparse, non-negative factorizationof a matrix of facial training images.
 9. The method of claim 1 whereinat least one of the non-humanly-perceivable sets of data representingfacial images of individuals whose identities are known is obtained by amethod comprising performing a sparse, non-negative factorization of amatrix of humanly-perceivable facial training images.
 10. A method forperforming facial image recognition comprising: (a) transforming, usinga computer system, a facial image of a subject which ishumanly-perceivable into a facial image which is nothumanly-perceivable; and (b) classifying, identifying, or classifyingand identifying the facial image that is humanly-perceivable bycomparing, using a computer system, the facial image that is nothumanly-perceivable with a library of facial images that are nothumanly-perceivable, the library of facial images being for subjectswhose classification, identity, or classification and identity is known.11. The method of claim 10 wherein the transformation is a sparse,non-negative transformation.
 12. The method of claim 11 wherein thetransformation is performed using a predetermined transformation matrixobtained by a method comprising using a computer system to perform asparse, non-negative factorization of a matrix of facial trainingimages.
 13. The method of claim 11 wherein at least one of thenon-humanly-perceivable facial images in the library is obtained by amethod comprising transforming, using a computer system, ahumanly-perceivable facial image of an individual whose classification,identity, or classification and identity is known into anon-humanly-perceivable facial image.
 14. The method of claim 13 whereinthe transformation is a sparse, non-negative transformation.
 15. Themethod of claim 14 wherein the transformation is performed using apredetermined transformation matrix obtained by a method comprisingusing a computer system to perform a sparse, non-negative factorizationof a matrix of facial training images.
 16. The method of claim 11wherein at least one of the non-humanly-perceivable facial images in thelibrary is obtained by a method comprising performing a sparse,non-negative factorization of a matrix of humanly-perceivable facialtraining images.
 17. A non-transitory computer readable mediumcomprising a set of data for an individual for use in performing facialimage recognition, said set of data protecting the privacy of theindividual by not producing a humanly-perceivable image of theindividual when the data is displayed as a two-dimensional array. 18.The non-transitory computer readable medium of claim 17 wherein theindividual is an individual to be recognized.
 19. The non-transitorycomputer readable medium of claim 17 wherein the individual is anindividual whose identity is known.
 20. The non-transitory computerreadable medium of claim 17 comprising a plurality of sets of data for aplurality of individuals whose identities are known, at least one set ofdata for each individual, each of said sets of data not producing ahumanly-perceivable image of the individual when the data is displayedas a two-dimensional array.